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ABSTRACT 
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The  Government  Accounting  Office  (GAO)  has  stated  that  in- 
sufficient attention  is  given  to  Human  Factors  Engineering 
(HFE)  in  the  design  of  systems  during  the  Weapons  Acquisition 
Cycle  (WAC) .   According  to  GAO  these  inadequacies  have  adverse- 
ly impacted  our  military  capabilities  and  wasted  lives  and  mil- 
lions of  dollars.   A  myriad  of  handbooks,  manuals  and  standards 
exist  which  provide  detailed  guidelines,  criteria,  and  test 
plans  for  conducting  HF  T  §  E  which  remain  unused  because  their 
technological  level  is  beyond  the  average  user.   The  need  for 
basic  training  in  HFE  has  been  clearly  identified.   A  cost  ef- 
fective vehicle  to  bridge  this  gap  in  conceptual  knowledge  has 
been  developed  in  the  form  of  an  HFE  Self -Paced  Course.   As 
coordinated  with  the  course's  sponsor,  an  internal  review  and 
trial  run  were  conducted,  via  this  thesis,  to  assess  its  po- 
tential effectiveness.   According  to  the  results,  this  course 
has  proven  its  capability  to  provide  the  stimuli  necessary  for 
the  transfer  of  basic  knowledge  and  skills  in  HF  T  5  E.   Addi- 
tionally, the  value  of  the  job  tasks  identified  in  the  course's 
terminal  objectives  were  substantiated.   The  conclusions  pro- 
vided in  this  study  are  intended  to  encourage  further  course 
development  through  incorporation  of  the  recommendations  out- 
lined.  Ultimately,  this  would  lead  to  its  validation  and 
implementation  into  the  instructional  system  of. the  military. 
Implementation  of  such  a  basic  course  could  be  a  major  step 
toward  increased  integration  of  HF  T  5  E  during  the  WAC. 
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I.   INTRODUCTION 

There  are  many  definitions  available  for  the  term  "human 
factors".   Simply  stated,  the  major  concern  of  all  people  in- 
volved with  human  factors  is  that  of  providing  equipment  and 
systems  which  people  can  use.   Included  in  the  various  facets 
of  human  factors  are  design,  test  and  evaluation. 

Technology  has  advanced  rapidly  during  the  last  decade. 
The  development  of  the  people  who  must  make  use  of  these 
"state-of-the-art"  technological  wonders  has  not  kept  abreast 
of  the  rapid  growth  in  this  field.   For  this  reason,  there 
has  been  increased  attention  to  the  human  factors  aspects  of 
the  new  systems.   Design  engineers  and  evaluators  must  be 
cognizant  of  this  fact,  so  that  rather  than  designing  equip- 
ment for  fellow  technical  wizards'  use,  they  must  take  into 
consideration  the  attributes  and  capabilities  of  the  ulti- 
mate user.   [Ref.  1:  pp.  1-27]   Referred  to  in  U.  S.  Navy 
terms,  the  requirement  is  to  make  it  "sailor-proof".   Be- 
cause the  majority  of  sailors  or  non-prior  service  enlisted 
personnel  as  measured  by  the  Armed  Service  Vocational  Bat- 
tery (ASVAB)  (Table  1)  fall  into  the  lower  mental  categories 
[Ref.  2:  p.  281],  this  requirement  is  of  major  concern  when 
implementing  new  systems  or  equipment  for  their  operation. 

Failure  of  deployed  systems  are  often  caused  by  human 
induced  errors.   There  are  indications  that  the  percentage 
of  failures  due  to  human  ineptitude  or  poor  human 
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reliability-'-  may  be  quite  high.   The  increasingly  compli- 
cated nature  of  modern  military  systems  together  with 
shortages  of  qualified  military  personnel  suggests  that 
human- induced  errors  both  in  operation  and  maintenance  of 
systems  will  increase  unless  more  attention  is  given  to 
this  problem  in  the  design  and  development  phases  of  the 
acquisition  process.   [Ref.  1:  p.  27] 


Table  1 

Mental  Category  of  Non-Prior  Service 
Enlisted  Accessions  for  1978 

Mental    Population  Marine  Air 

Category  Percentile  POD  Army  Navy  Corps   Force 

I  +  II    65  -  100    34   26    38    29     45 

III  31  -  .  64    61   63    60    68      55 

IV  10  -   30     5   11     3     4      0 


The  Department  of  Defense  [Ref.  1:  p.  34]  is  aware  of 
this  problem  and  concurs  that  effective  action  must  be  taken 
to  correct  it. 

Existing  military  instructions  and  directives  [Ref.  4] 
state  that  human  factors  engineering  must  be  integrated  into 
the  standard  test  and  evaluation  plans  of  all  military  sys- 
tems.  Command  level  documents  [Ref.  5]  include  statements 


•^Human  reliability  as  used  here  is  defined  as  the  probabil 
ity  that  human  error  by  either  the  operator  or  the  maintainer 
will  not  cause  a  system  failure  or  malfunction.   The  concept 
of  system  performance  employed  by  the  HFE  Self-Paced  Course 
is  not  concerned  with  human  error  that  does  not  cause  system 
failure.   The  evaluator  concentrates  on  overall  system  per- 
formance.  "The  human  is  an  integral  part  of  a  system  and 
therefore  his/her  performance  becomes  a  system  component  for 
evaluation.  [Ref.  3:  pp.  3-14] 
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which  say  that  human  factors  aspects  of  equipments  and  sys- 
tems will  be  assessed.   All  too  frequently,  however,  test 
directors  design  the  evaluation  plan  without  human  factors 
tests,  or  with  human  factors  testing  scheduled  on  a  "not-to- 
interfere  basis"  or  without  considering  the  basic  "...char- 
acteristics of  personnel  who  will  operate  and  maintain  the 
system  when  it  is  eventually  deployed,  i.e.,  muscular 
strength  and  coordination,  body  dimensions,  perceptions  and 
judgment,  sensory  capacities,  native  skills  and  capacity  to 
learn  new  skills,  optimum  workload,  basic  requirements  for 
comfort,  safety,  and  freedom  from  environmental  stress." 
[Ref.  1:  p.  29] 

An  additional  problem  arises  when  the  evaluators  become 
involved  too  late  in  the  development  and- acquisition  cycle-- 
the  "die  have  been  cast";  the  changes  required  are  too  ex- 
pensive in  terms  of  time  and/or  dollars.   Thus  test  directors 
are  forced  into  the  world  of  trade-off s--instead  of  receiving 
the  best,. the  user  will  get  whatever  the  least  costly  compro- 
mise had  to  offer  or  whatever  "hurt"  the  least.   This  often 
occurs  because  of  the  evaluator's  inexperience  with  the  design, 
test  and  evaluation  processes  and/or  lack  of  guidance,  train- 
ing, and  education  in  the  field  of  human  factors. [Ref  1:  p. 
10,  21,  31] 

Various  elements  of  the  Department  of  Defense  and  civilian 
authors  have  attempted  to  develop  documents  which  provide  hu- 
man factors  engineering  test  and  evaluation  guidelines.   Some 
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of  these  guidelines  tell  the  planner  when  the  evaluator 
should  first  become  involved,  others  provide  evaluation  check- 
lists and  criteria  against  which  to  measure.    A  variety  of 
documents  even  tell  how  to  perform  the  tests.   From  discus- 
sions held  with  various  members  in  the  test  and  evaluation 
field,  indications  are  that  most  of  these  existing  references 
lie  in  sad  disuse.   They  add  that  for  all  the  expertise  in- 
volved in  their  development,  no  one  has  to  date,  provided 
the  one  link  that  would  put  these  documents  into  high  demand: 
basic  training  and  education  in  human  factors.   Valuable 
documents  such  as  Human  Factors  Test  and  Evaluation  Manual 
(HFTEMAN) ,  Human  Factors  Engineering  Data  Guide  for  Evalua- 
tion (HEDGE),  and  Mission  Operability  Assessment  Technique 
(MOAT),  to  name  just  a  -few,  require  a  basic  knowledge  of 
human  factors  for  their  effective  use.   Consistent  with  hu- 
man nature,  it  is  easy  to  put  off,  ignore,  or' down  play  the 
importance  of  those  things  which  are  not  easily  understood. 

Having  recognized  the  need  for  basic  training  in  human 
factors,  the  question  then  becomes  one  of  implementation. 
The  means  of  instruction  must  take  into  account  cost  effec- 
tiveness, and  instruction  standardized  so  that  all  intended 
users  may  benefit.   During  the  1970 's  and  1980 's,  the  use  of 
the  Personalized  System  of  Instruction  (PSI)  has  gained  in 
popularity  and  credibility  [Ref  6:  pp.  1-9]  because  of  its 
economical  feasibility  [Ref  6:  pp.  11-16]  and  attention  to 
different  individual  learning  styles  [Ref.  6:  pp.  73-75]. 
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Critical  aspects  of  the  previously  heretical  concepts  of  PSI 
have  been  undergoing  intensive  study  [Ref.  6:  pp.  1-9;  Ref.  7 
pp.  587-592]  with  proven  positive  results.   The  major  success 
or  failure  of  PSI  is  greatly  dependent  upon  its  design  con- 
siderations.  The  subject  matter  expert  usually  knows  what 
content  must  be  included,  but  often  times  lacks  the  famili- 
arity and  understanding  necessary  for  PSI  development  to  give 
all  students  an  equal  opportunity  to  master  the  content. 
Frequently,  the  problem  lies  in  its  method  of  presentation 
and  delivery.  [Ref.  8:  pp.  40-43]   Adherence  to  the  basic 
requirements  for  a  PSI  design  and  development  can  produce  an 
extremely  effective  course  [Ref.  9:  pp.  165-170]. 

Progressing  with  the  advances  of  modern  education  and 
seeking  cost  effectiveness  in  delivering  instruction,  the 
HFE  Self-Paced  Course  designers  elected  to  develop  a  PSI  for 
use  with  its  small  and  specialized  enrollment.   The  U.  S. 
Navy  has  formally  recognized  procedures  and  agencies  for  PSI 
development,  design,  implementation  and  evaluation  [Refs.  10, 
11]. 

It  is  the  purpose  of  this  thesis  to  investigate  a  human 
factors  engineering  self-paced  course  which  is  anticipated 
to  provide  the  missing  link--education  and  training  in  basic 
human  factors.   Performance  of  this  investigation  will  be 
accomplished  in  three  phases: 
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1.  Development  o£  an  evaluation  plan  for  a  self -paced 
course  in  human  factors  in  accordance  with  CNET  and  CNTECHTRA 
directives. 

2.  Implementation  of  the  evaluation  plan  in  a  realistic 
environment,  and 

3.  Evaluation  of  the  Human  Factors  Engineering  Self- 
Paced  Course. 

A  description  of  a  preliminary  draft  of  the  Human  Factors 
Engineering  Self-Paced  Course  is  provided  in  Chapter  II.   The 
evaluation  plan  used  for  the  effectiveness  assessment  of  the 
course  is  discussed  in  Chapter  III.   Methodology  and  results 
are  addressed  in  Chapter  IV.   Chapter  V  presents  this  author's 
conclusions  and  recommendations.   A  final  summary  is  given 
in  Chapter  VI. 
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II.   THE  HUMAN  FACTORS  ENGINEERING  (HFE)  SELF-PACED  COURSE 

The  Human  Factors  Engineering  Self-Paced  Course  is  clas- 
sified as  a  Self-Teaching  Exportable  Package  (STEP)  by  . 
NAVEDTRA  106A.   As  such,  this  course  is  intended  to  be  self- 
supporting  with  minimal  student  contact  required  with  the 
distributing  command. 

Once  the  course  is  implemented  into  the  system,  any  mili- 
tary member  may  order  the  course  materials  from  the  sponsor- 
ing command  or  through  standard  military  supply  channels. 
Upon  receipt  of  the  course  materials,  the  student  may  proceed 
with  the  lessons,  working  at  home  or  during  free  time  while 
"on  the  job".   A  recommended  schedule  is  provided  to  each 
student  as  a  guideline  for  meaningful  and  timely  completion 
of  the  course  materials.   Thus  the  participant  proceeds  through 
the  lessons  on  a  self-paced  basis  while  continuing  to  perform 
duties  and  tasks  at  the  parent  command. 

This  self -paced  course  has  two  formats:   (1)  a  set  of 
written  manuals,  and  (2)  a  computer-based  parogram  planned  to 
be  available  on  the  Advanced  Research  Project  Agency  (ARPA) 
network  on  a  time-share  basis.   The  total  cost  of  either  pro- 
gram is  estimated  to  be  considerably  less  in  dollars  (travel 
and  per  diem),  man-hours  required  for  training  attendance, 
and  man-hours  lost  from  performing  on-the-job  tasks  and 
duties  while  attending  training  than  sending  personnel  to  off- 
site  human  factors  training  facilities.   This  evaluation  was 
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conducted  using  the  written  manuals  because  the  computer 

2 
version  was  not  accessable  at  the  time. 

Because  the  written  materials  are  so  extensive,  they  are 

not  provided  as  part  of  this  thesis.   However,  excerpts  and 

summarizations  taken  from  the  course  materials  covering  the 

course  purpose,  design,  description,  learning  objectives  and 

content  are  provided  in  the  remainder  of  this  section. 

A.  PURPOSE 

This  self-paced  course  was  developed  to  provide  HFE 
practioners  with  an  awareness  of  its  facts,  principles  and 
issues.   The  course  designers  have  intended  that  upon  comple- 
tion of  their  course  of  study,  each  participant  should  "think 
about  human  factors  whenever  they  are  doing  their  job". 
Therefore,  its  purpose  is  to  bridge  the  gap  between  Depart- 
ment of . Defense  (DOD)  directives  and  instructions  on  HFE  de- 
sign, test  and  evaluation  and  the  knowledge  level  of  the 
novice  HFE  evaluator. 

B.  COURSE  DESIGN  AND  STRUCTURE 

The  HFE  Self-Paced  Course  is  a  STEP  [Ref.  10:  p. 140], 
consisting  of  a  student  supplement,  40  lessons,  and  various 


2 
The  computer  version  has  been  written  but  time  and  fi- 
nancial constraints  made  it  impossible  to  evaluate.   Portable 
computer  terminals  were  required  for  distribution  at  test 
sites  and  these  were  not  available.   The  program  had  also  not 
been  entered  on  to  the  ARPA  network  for  systemwide  access. 
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military  standards  (MIL-STDS) ,  handbooks  (HDBKs)  and  docu- 
ments.  The  course  is  intended  to  provide  the  student  with 
the  same  sort  of  instruction  that  would  be  found  if  she/he 
were  the  only  student,  of  an  instructor.   The  written  manuals 
consist  of  eight  lesson  books,  each  containing  five  lessons. 
The  student  supplement  is  to  be  used  in  concert  with  either 
the  written  or  computer  generated  format. 

The  basic  course  has  been  divided  into  three  modules/ 
sections  (major  subject  areas],  the  first  contains  20  lessons, 
the  second  15  lessons  and  the  third  two  lessons.   Practical 
application  is  required  in  lessons  38  and  39  while  a  full 
course  review  is  provided  in  the  last  lesson.   Each  lesson 
consists  of  a  series  of  small  blocks  of  information.   A 
question  is  asked  at  the  end  of  each  block  to  check  compre- 
hension and  progress,  providing  immediate  feedback  to  the 
student.   Each  question  has  several  responses  from  which  to 
choose  and  the  student  must  select  the  answer  believed  to  be 
best.   The  student  is  then  either  directed  to  a  page  from  the 
lesson  book  or  presented  with  a  computer  image  which  contains 
a  comment  on  the  selected  answer  stating  whether  it  is  cor- 
rect or  incorrect,  and  why.   After  providing  remedial  infor- 
mation following  an  incorrect  selection,  the  student  is 
directed  to  choose  another  answer.   If  the  answer  is  again 
incorrect,  the  cycle  continues  until  the  student  selects  the 
correct  answer.   In  the  written  format,  the  order  of  the  ma- 
terial, questions  and  responses  is  scrambled  so  that  it  is 
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not  possible  to  skim  the  pages  o£  each  lesson  book  and  there- 
by skip  the  progress  checks.   Design  o£  the  computer  version 
also  eliminates  "skimming".   There  are  no  "End  of  Module/ 
Section  or  Lesson  Progress  Tests", 

•  Each  lesson  is  developed  in  a  story  line.   The  central 
character  is  a  junior  officer  (Lieutenant)  who  is  unskilled 
and  untrained  in  human  factors.   In  the  beginning,  the  lieu- 
tenant is  characterized  as  having  a  negative  attitude  towards 
human  factors.   As  the  course  continues,  various  events  cause 
the  officer  to  become  a  proponent  of  human  factors.   It  is 
intended  that  the  student  learn  from  the  numerous  mistakes 
of  the  ficticious  Lt.  I.  M,  Eager  and  the  guidance  provided 
by  his  HFE  mentor  and  boss,  Capt.  B.  Smart. 

In  addition  to  the  data  presented  throughout  the  lessons, 
a  student  supplement  has  been  provided.   As  the  student  pro- 
gresses through  each  lesson,  he/she  is  asked  to  refer  to  the 
supplement.   The  supplement  provides  the  student  with  graphs, 
photos  and  charts,  some  practical  work  (mathematics  and 
drawings),  and  "nice-to-know"  information. 

The  course  has  been  designed  with  the  intent  that  each 
should  take  no  longer  than  one  hour  to  complete,  with  an 
average  completion  time  of  35  -  40  minutes  and  should  be  done 
on  a  daily  basis.   Therefore,  maximum  total  completion  time 
should  not  exceed  40  hours,  or  40  working  days  if  one  lesson 
is  finished  per  day.   It  is  anticipated  that  the  computerized 
version  would,  in  all  probability,  reduce  the  completion  time 
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for  each  lesson,  as  it  eliminates  the  procedure  of  flipping 
through  the  lesson  books  searching  for  progress  check  re- 
sponses and  new  material.   The  computer  locates  the  response 
and  displays  it  automatically  for  the  student. 

In  addition  to  the  aforementioned  course  materials  vari- 
ous MIL-STDs,  handbooks  and  documents  are  provided  to  the 
student  and  their  usage  is  addressed  throughout  the  course. 
Some  are  required  for  successful  course  completion^,  while 


3 
Documents  required  for  successful  course  completion: 

a.  MIL-H-46885B.   Human  Engineering  Requirements  for 
Military  Systems,  Equipments  and  Facilities.  Janu- 
ary 19  79. 

b.  MIL-STD-1472C.   Human  Engineering  Design  Criteria 
for  Mlitary  Systems,  Equipments  and  Facilities. 

c.  MIL-STD-1747B.   Noise  Limits  for  Army  Materiel. 
March  1975. 

d.  MIL-HDBK-759.   Human  Factors  Engineering  Design  for 
Army  Materiel.   March  19  75 

e.  Army  Regulation  602-1.   Personnel-Materiel  Systems: 
Human  Factors  Engineering  Program.   U.  S.  Army 
Headquarters ,  Washington^  D.  C. ,  June  19  76. 

f.  Technical  Memorandum  29-76.   Guide  for  Obtaining  and 
Analyzing  Human  Performance  Data  m  a  Materiel  De- 
velopment Project.   Berson,  B.  IT.    and  Crooks,  W.  H.  , 
W.    St  Army  Human  Engineering  Laboratory,  Aberdeen 
Proving  Ground,  Md. ,  September  1976. 
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4 
others  are  referred  to  as  additonal  resources  .   While  the 

basic  package  of  reference  documents  provided  with  the  course 
materials  must  be  returned  along  with  the  lesson  books  to  the 
issuing  agency,  they  may  be  ordered  through  the  standard  mil- 
itary supply  system.   A  brief  summary  of  each  of  these  re- 
sources is  provided  in  Appendix  A. 

C.   COURSE  PRESENTATION 

The  course  materials  are  presented  in  a  manner  which 
allows  the  student  to  complete  the  course  of  instruction 
without  the  aid  of  an  on-site  instructor.   The  student  sup- 
plement is  intended  to  be  retained  by  the  student  and  used 


4 
Documents  referred  to  but  not  required  for  course  com- 
pletion: 

a.  MIL-STD-721B.   Definitions  of  Effectiveness  Terms 
for  ReliabilityT"  Maintainability ,  Human  Factors,  "and 
Safety.   August  19  66. 

b.  FED-STD-595.   Colors.   26  August  1964. 

c.  Holshouser,  E.  L. ,  Guide  to  Human  Factors  Engineer- 
ing General  Purpose'  Test  Planning  (GPTPT^   Pacific 
Missile  Test  Center,  Pt.  Mugu,  CaT 

d.  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN) , 
Vols.  Tl  II,  and  III.  Pacific  Missile  Test  Center, 
Pt.  Mugu,  Ca. 

e.  Human  Factors  engineering  Data  Guide  for  Evaluation 
(HEDGE) .   U.  S.  Army  Test  and  Evaluation  Command. 
20  December  1977. 

f .  Mission  Operability  Assessment  Technique:   A  System 
Evaluation  Methodology  (MOAT).   October  1979.  TP-79- 
31.   Pacific  Missile  Test  Center,  Pt.  Mugu,  Ca.   Lt . 
W.  R.  Helm  and  M.  L.  Donnell,  Ph.D. 
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as  a  "stand-alone"  resource  and  reference  document  following 
course  completion.   The  student  may  contact  the  issuing  agency 
in  the  event  additonal  help  is  required. 

D.  COURSE  LEARNING  OBJECTIVES 

In  accordance  with  the  recommendations  of  NAVEDTRA  106A, 
a  list  of  performance  requirements  was  collected  by  course 
designers  after  consultation  with  human  factors  engineering 
test  and  evaluation  personnel.   These  were  then  restated  into 
task  requirements  (Appendix  B)  and  forwarded  to  the  civilian 
contractor  who  was  assisting  in  the  course  development.   The 
course  learning  objectives  (Appendix  C)  were  then  developed 
from  this  list  of  task  requirements.   No  module  or  lesson 
learning  ojbectives  were  developed. 

E.  COURSE  CONTENT 

As  described  in  the  introduction  of  the  Student  Supple- 
ment, the  course  is  divided  into  three  major  content  sections 
(modules):   (1)  human  capabilities  and  limitations,  (2)  human 
integration  with  the  system,  and  (3)  "Human  Factors  in  the 
Military". 

Module  I  (Lessons  1-20)  investigates  the  history  and  con- 
tinued need  for  human  factors  engineering.   Physiological  ca- 
pabilities and  limitations  of  human  beings  and  their 
incorporation  into  the  proper  design  and  controls  and  displays 
are  addressed.   Also  included  is  a  discussion  of  the  human's 
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interactions  with  the  surrounding  environment  and  the  effects 
it  may  have  upon  his/her  performance. 

Module  II  (Lessons  21-35)  discusses:   (1)  the  role  of  the 
human  engineer  in  various  aspects  of  analysis,  i.e.,  systems, 
cost,  task  analysis  and  etc.,  (2)  selection  and  training  of 
personnel;  and  (3)  experimental  techniques  and  statistical 
concepts . 

Module  III  (Lessons  36-37)  focuses  on  human  factors  organ- 
izations, documents,  and  future  applications.   To  provide 
students  with  a  practical  application  of  what  has  been  pre- 
sented in  the  course,  lessons  38  and  39  present  a  "real  world" 
problem  for  solution.   Finally,  in  lesson  40,  a  more  typical 
review  of  the  entire  course  is  conducted. 

The  outline  of  the  Module  and  lesson  topics  is  provided 
in  Appendix  C. 
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III.   EVALUATION  PLAN 


The  validation  process  is  a  critical  step  in  the  devel- 
opment of  lesson  materials.   The  question  to  be  asked  is: 
"How  can  we  be  certain  that  the  instruction  works?"   The 
only  acceptable  way  to  do  this  is  to  measure  student  per- 
formance.  If  the  students  learn  the  specified  skill, 
task,  or  knowledge  as  a  result  of  the  instruction,  then 
it  is  valid;  if  the  students  do  not  learn  as  specified  in 
the  learning  objective,  the  instruction  is  not  valid  and 
must  be  revised.   Validation  can  be  compared  to  the  test- 
ing process  on  a  new  piece  of  equipment.   If  particular 
components  of  the  equipment  malfunction  repeatedly,  then 
this  indicates  a  problem  that  must  be  corrected  if  the 
equipment  is  to  perform  properly.   By  the  same  token,  if. 
the  students  fail  to  master  a  particular  segment  of  in- 
struction, a  problem  must  be  corrected.   [Ref.  11] 


To  define  the  strategy  used  to  assess  the  potential  effec- 
tiveness of  the  instructional  materials  for  the  HFE  course, 
an  evaluation  plan  was  developed  (Appendix  E) ,  in  accordance 
with  the . guidelines  provided  in  NAVEDTRA  llOA.   The  evaluation 
plan  provided  in  this  thesis  combined  several  of  the  valida- 
tion steps  prescribed.   These  combinations  were  made  because 
(1)  no  validation  plan  had  been  developed  by  the  course  spon- 
sors, (2)  no  previous  evaluation  of  the  course  materials  had 
been  conducted,  and  (3)  the  course  was  in  an  advanced  state 
of  development.   While  the  computer  version  has  been  written, 
it  is  not  available  on  a  commonly  accessible  system.    The 
evaluation  was,  therefore,  developed  using  the  set  of  written 
manuals . 
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A.   HOW  TO  CONDUCT  VALIDATION 

Valid-ation  is  accomplished  by  completing  the  following 
steps  [Ref .  11] : 

1.  Interval  Review.  As  the  rough  draft  materials  are 
evaluated,  the  following  questions  should  be  asked: 

a.  Is  the  content  accurate? 

b.  Is  the  material  presented  in  a  logical  fashion? 

c.  Does  the  lead-in  information  motivate  the  student 
to  pursue  the  material? 

d.  Do  the  teaching-learning  activities  encourage 
productive  learning? 

e.  Is  the  material  written  in  a  .manner  to  allow 
maximum  student  participation? 

f.  Is  there  an  opportunity  for  review  and  practice? 

g.  How  effectively  does  the  material  teach  behaviors 
specified  in  the  learning  objectives? 

h.   Does  the  test  measure  the  behavior  as  specified 
in  the  learning  objectives? 

The  internal  review  does  not  determine  validity  of  the 
course  material.  It  does,  however,  identify  problem  areas 
and  suggest  alternatives. 

2.  Individual  Trials.   Informal,  individual  trials  on 
rough  draft  materials  for  each  lesson  will  take  place  to 
identify  gross  deficiencies  in  the  lesson  materials  "before" 
expensive  final  draft  materials  are  produced.   In  other 


29 


words,  these  trials  should  help  identify  where  more  practice 
is  needed,  instruction  is  adequate,  and  whether  instructor 
guides  or  student  study  booklets  are  complete.   Since  this  is 
probably  the  first  time  students  will  actually  take  the  test 
items  developed  for  a  lesson,  the  informal  trial  may  also 
identify  problem  test  items. 

The  individual  trials  should  be  conducted  as  follows: 

a.  Administer  Pretests.   Pretests  are  given  to  stu- 
dents to  determine  how  much  they  know  before  being  exposed 
to  the  instruction.   The  scores  made  on  the  pretest  are  then 
compared  with  scores  made  on  post  tests  to  ascertain  the  ef- 
fect instructional  materials  had  on  student  performance. 

Tests  developed  for  validation  purposes  are  com- 
posed of  the  content  items  that  address  both  enabling  and 
terminal  objectives.   For  validation  purposes,  however,  it 
may  be  helpful  to  add  test  items  to  both  pre  and  post  tests 
which  would  be  diagnostic  in  nature  and  therefore  help  to 
locate  problem  spots  in  the  instructional  materials. 

b.  Administer  Instructional  Materials.   Observe  the 
students'  performances  as  they  complete  the  lesson  and  record 
their  completion  time,  the  questions  asked,  and  any  difficul- 
ties observed. 

c.  Administer  Post  Test  Items.   This  will  measure 
mastery  of  the  content  by  the  students.   The  results  will  be 
used  to  identify  strengths  and  weaknesses  in  the  course  and 
to  provide  a  basis  for  making  revisions. 
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d.  Administer  Student  Feedback  Sheets.   This  is  the 
student's  opportunity  to  suggest  how  to  make  instructional 
materials  more  effective.   As  siich,  the  feedback  sheet  should 
be  structured  so  that  students'  responses  are  channeled,  yet 
flexible  so  that  open  ended  responses  are  allowed. 

e.  Revise  the  Instructional  Materials.   Evaluate  all 
data:   test  scores  and  instructor  and  student  comments.   Then 
make  the  revisions  required.   When  all  major  deficiencies  have 
been  corrected  and  revalidated,  the  individual  trials  are 
concluded. 

3.   Group  Trials.   A  minimum  of  six  students  should  be 
used  for  each  group  trial.   As  with  individual  trials,  it  is 
very  important  to  select  students  that  are  representative  of 
the  target  population  and  that  have  the  proper  prerequisites 
for  the  experimental  group.   The  steps  for  conducting  group'  . 
trials  are  the  same  as  were  followed  during  the  individual 
trials  with  one  major  exception:   this  time,  there  is  only 
one  observer  to  six  or  more  students.   As  students  proceed 
through  the  instruction,  they  are  not  interrupted;  they  con- 
tinue through  it  as  would  a  normal  class.   Only  upon  comple- 
tion of  the  lesson  materials  may  students  be  asked  such  ques- 
tions as  why  they  missed  certain  items,  did  the  instructional 
materials  maintain  their  interest,  were  there  enough  practice 
items . 


31 


4,   Operational  Validation.   Validation  of  a  new  or  revised 
course  will  be  conducted  in  the  operational  environment  using 
the  normal  student  population.   The  reasons  for  conducting 
the  operational  validation  are: 

a.  Instructional  materials  must  be  evaluated  as  an 
integral  part  of  a  total  system.   Until  now,  individual  and 
group  trials  validated  portions  of  instructional  material  in 
an  isolated  environment. 

b.  Analysis  of  data  from  this  larger  sample  will 
provide  a  solid  basis  for  final  revision  and  refinement  of 
the  course.   Data  gathered  at  this  point  will  provide  feed- 
back concerning  the  adequacy  of  the  learning  analysis,  learn- 
ing objectives,  criterion  tests,  and  instructional  materials. 
If  students  fail  to  meet  the  standards  of  the  terminal  objec- 
tives, reassessment  of  each  of  these  procedures  and  products 
of  the  course  design  and  development  processes  shall  be  scru- 
tinized. 

c.  An  operational  trial  provides  an  opportunity  to 
work  out  administrative,  equipment,  facility,  or  any  other 
implementation  problems  which  may  cause  trouble  later. 

The  steps  for  completing  operational  validation  are: 
(1)   Review  of  Material  by  Functional  Command. 
The  functional  command  will  review  a  cross-section  of  all 
instructional  materials  for  content,  technical  accuracy,  and 
format.   These  will  include  instructor  guides,  student  study 
guides,  remediation  guides,  media,  student  study  booklets  and 
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laboratory  guides.   Discrepancies  noted  by  the  func-tional 
command  will  be  discussed  with  the  curriculum  designer. 

(2)  Administer  Pretests,  Instructional  Materials, 
and  Post  Tests.   As  with  both  individual  and  group  trials,  it 
is  absolutely  necessary  to  have  students  with  the  appropriate 
entry  level;  without  this,  validation  results  are  contamina- 
ted. 

(3)  Administer  Student  and  Instructor  Feedback 
Sheets . 

(4)  Analyze  Results  and  Display/Present  Data  to 
Approval  Authority.   After  administering  all  the  tests,  in- 
struction, and  feedback  sheets,  the  results  must  be  analyzed 
and  displayed  for  submission  to  the  approval  authority.   Pri- 
or  to  operational  validation,  a  validation  standard  must  be 
set.   At  this  time,  the  validation  test  results  are  compared 
with  the  validation  standard.   If  the  standard  is  not  reached, 
the  curriculum  designer  should  look  carefully  at  the  instruc- 
tional materials,  equipments,  and  management  documents  to 
locate  the  problems  and  decide  on  corrective  action  to  be 
taken, 

B.   EVALUATION  PLAN 

The  format  used  for  the  Evaluation  Plan  was  modeled  after 
that  used  by  the  original  test  site  in  evaluating  equipment 
and  systems  [Ref.  5]  because  of  familiarity  with  selected 
test  site  supervisors.   Following  its  design  and  development. 
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other  military  branch  sites  requested  to  be  included  in  the 
evaluation.   A-  summary  of  the  evaluation  plan  follows: 

1,  Purpose,   The  purpose  of  the  evaluation  was  to  pro- 
vide an  assessment  of  the  potential  effectiveness  of  the  HFE 
Self-Paced  Course  and  its  potential  value  for  use  in  the  mil- 
itary test  and  evaluation  community.   The  evaluation  was  plan- 
ned in  cooperation  with  the  HFE  Laboratory  at  Pacific  Missile 
Test  Center,  Pt.  Mugu,  California. 

2.  Background.   The  HFE  Self -Paced  Course  was  developed 
to  satisfy  a  need  for  increased  awareness  and  more  indepth 
understanding  of  human  factors.   The  Government  Accounting 
Offic§  (GAO)  highlights  this  need  and  identified  a  defici- 
ency in  the  performance  of  various  weapon  systems 

...because  the  DOD  does  not  pay  enough  attention  to  lo- 
gistic support,  human  factors  and  quality  assurance  during 
the  design  phase  of  the  acquisition  process.   These  pro- 
blems deter  the  systems'  effectiveness  to  defend  our  coun- 
try in  case  of  war.   GAO,  therefore,  makes  recommendations 
to  improve  the  management  and  planning  of  ownership  con- 
siderations that  have  an  impact  on  the  effectiveness  of 
a  weapon  system.   [Ref.  1:  pp.  1-27] 

The  Human  Factors  Test  and  Evaluation  Manual  (HFTEMAN) 
Vols.  I,  II  and  III  was  distributed  to  various  government 
agencies  in  October  of  1976  by  Pacific  Missile  Test  Center. 
This  document  was  developed  to  provide  standardization  in 
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procedures,  testing  and  criteria  in  evaluating  human  factors. 
A  basic  knowledge  of  human  factors  terminology ,  principles 
and  concepts  was  necessary  for  its  most  effective  use.   This 
was  generally  lacking  and  therefore  HFTEMAN  was  not  utilized 
as  it  was  intended. 

The  HFE  Self-Paced  Course  was  developed  to  provide 
this  basic  knowledge.   It  has  not  been  evaluated  or  used  on 
a  trial  basis  in  any  portion  of  the  military  prior  to  this 
study. 

This  researcher  was  tasked  to  (1)  develop  an  evalu- 
ation plan  in  accordance  with  NAVEDTRAs  llOA  and  106A,  (2)  im- 
plement it,  conducting  the  test  in  a  realistic  environment, 
and  (3)  analyze  the  data  and  provide  results,  recommendations, 
and  conclusions.   Additional  taskings  and  responsibilities 
are  described  in  Appendix  E. 

3.  Prevalidation.   This  researcher  conducted  the  inter- 
nal review  with  the  assistance  of  content  and  curriculum 
development  subject  matter  experts. 

4.  Informal/Group  Trials/Operational  Trial. 

a.   Personnel  Selection.   Subjects  participating  in 
this  evaluation  were  drawn  from  four  military  commands  (two 
Navy  and  two  Air  Force)  whose  primary  mission  was  operational 
test  and  evaluation  of  equipment  and  systems  prior  to  their 
acquisition  by  DOD  for  service  use.   Personnel  were  to  parti- 
cipate on  a  volunteer  basis.   Students  selected  from  the 


35 


volunteers  were  to  be  representative  of  the  target  population 
which  would  be  ultimately  using  the  instructional  materials. 
The  following  personnel  prerequisites  were  established: 

(1)  Subjects  were  selected  from  various  warfare 
communities,  i.e.,  air,  surface,  subsurface  and  special  war- 
fare. 

(2)  No  constraints  regarding  rate,  rank,  speci- 
ality code,  grade,  educational  level  or  prior  experience  were 
imposed. 

(3)  Each  subject  was  required  to  be  actively 
involved  in  planning  and  performing  test  and  evaluation. 

b.   Testing  Constraints. 

(1)  Learning  Center  Instructor.   No  on-site 
instructors  were  available  at  any  of  the  test  sites,  nor 
were  they  required.   This  researcher  was  designated  as  the 
students'  contact  point,  tismporarily  fulfilling  the  future 
role  of  the  sponsoring  agency.   A  Command  Course  Supervisor 
(CCS)  was  identified  at  each  location  to  assist  this  re- 
searcher in  distributing  and  collecting  course- materials , 
tests,  feedback  sheets  (questionnaires),  and  recording  stu- 
dent questions  or  comments  and  demographic  data. 

(2)  Personnel  Selection.   While  HFE  designers 
and  evaluators  are  equally  important,  the  availability  of 
inexperienced  equipment  designers  needing  instruction  was 
extremely  limited  as  compared  to  those  in  equipment  evaluation, 
The  very  dynamic  and  mobile  nature  of  the  course  designers' 
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military  target  population  identified  the  intended  users  of 
the  course--U.  S.  military  operational  test  directors  involved 
in  whole  or  in  part  in  human  factors  evaluation.   The  original 
test  plan  called  for  the  use  of  volunteers  only,  however,  due 
to  limited  personnel  availability,  final  analysis  showed  that 
some  participants  had  been  assigned. 

(3)  Course  Materials.   Original  plans  allowed 
60  days  for  student  completion  of  all  40  lessons  of  the  HFE 
Self-Paced  Course.   As  a  result  of  delayed  procurement  of 
course  materials,  the  actual  evaluation  was  limited  to  30 
days  and  the  first  20  lessons  of  the  course. 

(4)  Course  Completion.  The  personnel  partici- 
pating in  this  evaluation  were  also  performing  their  normal 
jobs  as  test  directors.  Potential  conflict  was  anticipated 
with  their  normal  responsiblities  which  could  require  their 
absence  from  the  command  for  a  prolonged  period  of  time  and 
cause  interruption  in  course  progress  or  even  termination. 

(5)  Learning  Center.   No  area  was  available  at 
any  location  which  could  be  specifically  designated  for  stu- 
dent use.   Students  studied  in  offices,  libraries  and  at  home. 
This  was  congruent  with  the  course  design  and  the  intent  of 
the  study. 

(6)  Testing.   No  tests  had  been  developed  by  the 
sponsor.   The  tests  used  in  this  evaluation  were  developed  by 
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the  author  and  had  received  only  limited  pretesting  prior  to 
the  evaluation'because  of  time  constraints  and  therefore  had 
not  been  validated. 

(7)   Training.   No  training  beyond  that  offered 
by  the  HFE  Self-Paced  Course  was  conducted  during  the  opera- 
tional trial  run  of  this  study, 

5.   Judgment  Criteria.   Both  the  pretest  and  post  test 
were  designed  in  accordance  with  the  objectives  developed  by 
this  researcher  (Table  2).       , 

a.  Pretest.   The  pretest  was  used  to  measure  prior 
knowledge  of  the  course  content  possessed  by  entering  students 
The  pretest  was  composed  of  four  different  types  of  questions, 
each  being  assigned  a  specific  point  value  for  scoring: 

multiple  choice       1  point  per  answer  choice 
matching  1  point  each  response 

fill  in  the  blank.     2  points  each 
short  answer  3  and  5  points  each 

The  scores  c5f  the  Course  Group  we're  compared  to  those  of  the 
Control  Group  to  ascertain  entry  level  equivalency. 

b.  Post  Test.   The  post  test  was  administered  after 
completion  of  20  lessons  of  instruction  to  asses.s  whether: 
(1)  the  students  had  mastered  the  objectives  of  the  first 
module,  and/or  (2)  the  students  taking  the  course  knew  more 
about  human  factors  than  students  not  taking  the  course.   The 
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Table  2 

Learning  Objectives  Developed  (By  the  Researcher) 
For  Design  of  Pre  and  Post  Tests 

1.  Given  a  list  of  human  factors  terras,  write  a  definition 
for  each  in  your  own  words . 

2.  Given  a  list  of  specific  HFE  .probems  and/or  areas  of  con- 
cerns, identify  which  human  factors  references  should  be 
consulted  to  provide  solutions  and  guidelines. 

3.  Given  at  least  one  control/display  example,  use  the  ap- 
propriate human  factors  reference  to  determine  which 
features  need  to  be  redesigned. 

4.  Given  a  list  of  potential  sources  of  technical  informa- 
tion on  human  factors,  identify  the  general  purpose  of 
each. 

5.  Explain  in  your  own  words  the  importance  of  evaluating 
the  human  factors  aspects  of  systems  and  equipments. 

6.  Identify  the  human  factors  principles  which  must  be  ap- 
plied during  the  evaluation  of  a  workspace  and  control/ 
display  panel. 

7.  Given  the  four  phases  of  the  Weapons  Acquisition  Cycle, 
describe  the  HFE  specialist  role,  the  major  HFE  concerns 
and  objectives  of  testing  for  each  phase. 

8.  Given  the  mission  statement  and  the  three  functions  which 
must  be  performed  for  a  particular  system,  determine  the 
human  performance  requirements. 

9.  Given  a  list  of  10  factors  and  forces  which  affect  human 
performance j  identify  what  aspect  of  human  performance 
each  affects. 

10.  Given  a  list  of  10  factors  and  forces  which  affect  human 
performance,  identify  what  unit  of  measure  must  be  used 
to  assess  the  amount  of  protection  required  to  diminish 
their  adverse  effects  upon  the  human  operator. 

11.  Explain  in  your  own  words,  the  importance  of  performing 
a  task  analysis. 

12.  Given  a  choice  of  3  possible  task  analyses,  correctly 
identify  which  satisfies  the  definition  of  "task  analysis" 
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post  test  was  composed  of  the  same  type  o£  questions  and  the 
associated  point  values  as  the  pretest;  however,  the  propor- 
tionment  of  the  four  different  types  of  questions  was  dif- 
ferent so  as  to  reduce  "chance"  selection  of  correct  answers. 
The  post  test  was  designed  to  test  the  same  learning  objec- 
tives as  the  pretest.   The  post  test  scores  of  the  Course 
Group  were  compared  to  those  of  the  Control  Group  to  deter- 
mine whether  or  not  learning  had  occurred  as  indicated  by 
the  Course  Group *s  scoring  significantly  higher  than  the 
Control  Group. 

c.   Test  Items.   Each  test  item  was  "referenced"  to 
a  specific  terminal  objective.   Since  each  objective  was  a 
statement  of  a  "criterion"  for  student  performance,  these 
tests  for  objectives  were  identified  as  "criterion-refer- 
enced" tests.   The  test  items  were  developed  in  accordance 
with  the  guidelines  provided  in  NAVEDTRA  llOA.   The  questions 
and  answers  were  extracted  from  the  Human  Factors  Engineering 
Self-Paced  Course,  lesson  20  which  contained  review  material 
covering  lessons  1-10.   Student  responses  which  expressed 
the  same  answers  as  the  course  materials  were  given  credit. 
Partial  credit  was  awarded  where  possible.   Test  key  answers 
were  compared  to  the  answers  provided  by  selected  HFE  experts 
to  ensure  the  HFE  course  answers  were,  in  fact,  correct. 

Additonally,  to  help  identify  weaknesses,  test 
items  were  added  to  both  pre  and  post  tests  which  were 
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diagnostic  in  nature  and  therefore  would  help  locate  problem 
areas  in  the  instructional  materials.  Results  were  recorded 
for  each  test  item. 

d.  Hypothesis  for  Comparison.   If  there  was  no 
statistically  significant  difference  between  the  post  test 
scores  of  the  group  completing  the  first  module  and  the  scores 
of  the  Control  Group  which  did  not  take  the  course,  it  would 
indicate  that  the  course  materials  were  not  effective  and 
would  be  of  no  value  for  use  in  the  military. 

Further  analysis  would  then  be  conducted  on  both 
the  pretest  and  post  test  questions, 

e.  Statistical  Analysis.   The  statistical  analysis 
was  performed  on  the  pre  and  post  tests  and  was  intended  to 
establish  (1)  the  entry  level  equivalency  of  both  the  Course 
Group  and  the  Control  Group  based  on  the  pretest  scores  and 
(2)  to  establish  a  difference  in  the  post  test  scores  between 
the  two  groups.   The  results  of  the  post  test  scores  would 
therefore  provide  an  overall  statement  of  the  potential 
effectiveness  of  the  HFE  Self-Paced  Course. 

f.  Raw  data  for  the  pre-post  test  analysis  for  the 
operational  tryout  was  provided  to  this  author  in  conjunction 
with  students'  tests  and  is  presented  in  summary  form  (Appen- 
dix G) . 


41 


6.   Standards 

Criterion  to  establish  equivalency  between  both  the 
Course  Group  and  the  Control  Group  on  the  pretest  scores  had 
been  set  by  this  author.   There  must  be  no  statistically  sig- 
nificant difference  at  the  .025  level  (two-tail  t  test)  to 
determine  that  the  groups  are  equivalent  in  their  pre-entry 
level. 

Criterion  to  establish  that  learning  has  taken  place 
with  the  Course  Group  had  been  set  by  this  author.   There 
must  be  a  statistically  significant  difference  between  the 
Course  Group  and  the  Control  Group  post  test  scores,  at  the 
.025  confidence  level  (two-tail  t  test)  to  determine  that 
the  groups  are  not  equivalent  in  their  level  of  learning. 
To  validate  that  the  course  materials  were  able  to  success- 
fully provide  the  stimuli  for  knowledge  and  skills  transfer 
to  the  students  taking  the  course,  their  post  test  scores 
must  be  significantly  greater  than  the  Control  Group. 

Criterion  guidelines  were  established  by  CNTECHTRA 
for  acceptability  of  course  objectives,  design,  presentation, 
availability,  maintainability  and  supportability  in  NAVEDTRAs 
llOA  and  106A  and  are  summarized  by  this  researcher  in  Table 
3  and  described  in  greater  depth  in  Appendix  E. 

In  the  event  the  standards  are  not  met  following  the 
operational  tryout ,  instructional  materials  would  be  analyzed 
as  indicated  above. 
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7 .  Remediation 

Students  involved  in  the  operational  tryout  receive 
remediation  from  the  responses  to  the  Progress  Checks  follow- 
ing each  block  of  instruction  within  the  lesson  topics. 

8 .  Instructor  Training 

The  HFE  Self-Paced  Course  was  intended  for  use  by 
individuals  without  the  aid  of  an  on-site  instructor.   The 
intent  of  the  agency  sponsoring  the  course  is  to  provide  a 
point  of  contact  for  students  to  answer  any  questions  on  con- 
tent, procedures,  etc.,  once  the  course  is  approved.   There 
was,  however,  no  instructor  guide  available  during  this  study. 
If  none  is  to  be  developed,  the  intended  point  of  contact 
must  then  be  a  subject  matter  expert.   The  student  and  the 
expert  would  then  talk  through  any  problems  using  the  set  of 
student  course  materials. 

9 .  Student  and  Instructor  Feedback  Sheets  (Questionnaires) 
The  following  student.  Command  Course  Supervisor  and 

Curriculum  Development  Expert  feedback  questionnaires  were 
completed  and  returned  to  this  author  for  review  and  analysis. 

a.  Annex  (A)  of  Appendix  E  was  completed  by  each 
participant  in  the  evaluation  during  orientation. 

b.  Annex  (C)  of  Appendix  E  was  completed  by  each 
participant  prior  to  the  students  beginning  the  course. 

c.  Annex  (D)  of  Appendix  E  was  completed  by  each 
participant  after  the  Course  Group  had  completed  the  first 
20  lessons. 
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d.  Annex  (B)  of  Appendix  E  was  maintained  by  each 
Command  Course  Supervisor  (CCS)  throughout  the  evaluation 
period. 

e.  Annex  (G)  of  Appendix  E  was  completed  by  experts 
in  the  field  of  curriculum  development. 

f.  Annex  (F)  of  Appendix  E  was  completed  by  each 
Course  Group  participant  upon  completion  of  the  post  test. 

g.  Annex  (E)  of  Appendix  E  was  completed  by  each  CCS 
upon  completion  of  the  evaluation  period. 

10 .   Display  Format 

Test  analysis  results  are  displayed  as  Appendixes  F 
and  G  and  discussed  in  Chapter  IV  of  this  thesis. 

Analysis  of  the  questionnaires  are  displayed  in 
Appendix  H  and  discussed  in  Chapter  IV  of  this  thesis. 
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IV.   METHODOLOGY  AND  RESULTS 

For  this  study  52  participants  were  chosen  from  four 
test  sites,  each  of  whom  met  the  prerequisites  for  course 
entry  as  described  below.   These  52  participants  were  divided 
into  two  groups;  one  which  would  complete  the  course  (Course 
Group)  and  the  other  (Control  Group)  which  would  not  go 
through  the  course,  but  would  be  administered  the  pre  and  post 
tests.   Since  the  pre.  and  post  tests  were  to  be  structured 
differently,  the  post  test  scores  could  not  be  compared  to 
the  pretest  scores  to  measure  the  effectiveness  of  the  course 
in  providing  the  stimuli  for  transferring  knowledge  and  skills 
in  human  factors.   It  was,  therefore,  necessary  to  compare 
the  post  test  results  of  the  Course  Group  to  those  of  a 
Control  Group  who  had  not  completed  the  course  of  study. 
For  that  comparison  to  be  valid,  it  was  necessary  to  estab- 
lish an  equivalency  of  the  entry  levels  of  the  two  groups 
prior  to'  commencing  the  course  of  .instruction.   This  measure 
of  equivalency  included  data  gathered  from  demographic  infor- 
mation sheets  completed  by  each  participant  and  pretests  on 
the  subject  material. 

Of  the  original  26  students  beginning  the  course,  10  were 
unable  to  complete  the  20  lessons  due  to  conflicts  of  tasks 
imposed  by  their  job  responsibilities.   Therefore,  only  16 
post  tests  from  the  original  26  students  in  the  Course  Group 
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were  used  for  the  comparison.   All  26  post  tests  from  the  Con- 
trol Group  were  used.   A  breakdown  of  experiment  participants 
and  data  sources  is  provided  in  Table  4. 

The  comprehensive  examinations  used  for  the  pretest  and 
post  test  were  prepared  by  this  researcher  (Appendix  E,  An- 
nexes C  and  D)  and  structured  as  described  in  section  III  of 
this  thesis.   While  the  post  test  had  the  same  type  of  ques- 
tions as  the  pretest,  there  were  a  larger  number  of  short 
answer  and  essay  type  questions  on  the  post  test.   This  tend- 
ed to  eliminate  a  large  portion  of  guessing  and  required 
that  the  students  have  a  more  comprehensive  knowledge  of  the 
course  material.  [Ref.  11:  pp.  3-30]   Although  no  claim  is 
made  as  to  the  validity  of  these  examinations,  they  do 
represent  a  measure  of  student  knowledge  and  achievement  in 
relation  to  stated  learning  objectives  of  the  course. 

A.   DEMOGRAPHIC  DATA,  ANALYSIS  AND  RESULTS 

No  prerequisites  were  required  of  students  prior  to 
taking  the  HFE  Self -Paced  course.   A  prerequisite  established 
by  this  researcher  for  this  study  required  that  each  student 
be  actively  involved  in  test  and  evaluation  for  motivational 
purposes . 

A  Student  Information  (SIF)  sheet  was  distributed  to  and 
completed  by  each  participant.   Specific  information  included 
pay  grade,  educational  level,  time  stationed  at  the  test  sites, 
job  description  and  previous  courses  completed  in  human  factors 
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training.   This  demographic  data  for  each  participant  is  pro- 
vided in  Appendix  F.   A  comparative  breakdown  (Table  5)  be- 
tween both  groups  was  performed  to  specifically  identify 
demographic  characteristics  which  could  bias  the  results  of 
this  study. 

The  distribution  of  enlisted  personnel  and  officers 
between  both  groups  was  disproportionate  with  the  Course 
Group  having  50  per  cent  enlisted  and  38.5  per  cent  officers 
compared  to  the  Control  Group's  23.1  per  cent  enlisted  and 
65.4  per  cent  officers.   The  educational  level  of  the  two 
groups  was  also  unbalanced  with  80.8  per  cent  of  the  Control 
Group  having  college  and/or  postgraduate  degrees  compared 
to  the  57.7  per  cent  of  the  Course  Group.   Specifically  sig- 
nificant  are  the  percentages  of  those  having  only  high 
school  diplomas;  3,8  per  cent  of  the  Control  Group,  compared 
to  34.6  per  cent  of  the  Course  Group.   Additionally,  the 
Control  Group  had  an  average  of  35.9  months  experience  in 
test  and  evaluation  while  the  Course  Group  averaged  only  17.2 
months.   There  was  also  a  disparity  in  the  number  having 
participated  in  the  test  director's  course  offered  by  two  Of 
the  test  sites;  76.9  per  cent  for  the  Control  Group  and  53.8 
per  cent  for  the  Course  Group.   All  participants  in  this 
experiment  were  actively  involved  in  test  and  evaluation. 

These  results  caused  some  concern  that  test  scores  would 
be  biased  since  the  Control  Group  as  a  whole  seemed  more 
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experienced  in  performing  test  and  evaluation,  educated  to 
a  higher  level,  and- more  experienced  in  middle  management 
responsibilities  for  test  and  evaluation.   The  actual  re- 
sults of  the  pre  and  post  tests,  however,  did  not  substanti- 
ate this  concern. 

Additionally,  the  demographic  percentages  between  the 
original  26  and  the  final  16  members  of  the  Course  Group  who 
completed  all  20  lessons  vary  insignificantly. 

B.   PRE-POST  TEST  ANALYSIS  AND  RESULTS 

When  all  test  participants  had  completed  the  pretest,  the 
combined  scores  of  all  participants  (Appendix  F)  in  each  of 
the  two  groups  were  compared.   The  mean  score  for  the  Control 
Group  was  70,19  with  a  standard  deviation  of  14*85.   The 
mean  score  for  the  Course  Group  was  64.73  with  a  standard 
deviation  of  18.73.   The  following  pooled  variance  formula 
for  the  t-test  was  used  to  test  the  significance  of  the  dif- 
ference in  group  mean: 


x^  -  x^ 


t  = 


(n^  -  1)  s^^  +  (n2  -  1)  s^      \  /I   ^  1 


n^  +  n^  -  2 


Popham  and  Sirotnik  state  that  this  formula  will  result  in  a 
t-value  which  may  be  interpreted  with  more  degrees  of  freedom 
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than  other  t-formulas.  [Ref.  12:  p.  139]   Therefore,  since  a 
smaller  t-value  is  needed  to  reject  a  given  null  hypothesis 
when  a  greater  number  o£  degrees  o£  freedom  are  used,  this 
formula  produces  a  t-test  that  is  more  likely  to  be  signifi- 
cant and,  as  a  result,  it  is  a  more  powerful  test. 

Hypothesis  one  assumes  -^hat  the  students  composing  the 
two  groups  do  have  an  equivalent  knowledge  level  prior  to  the 
entry  of  the  experiment  and,  therefore,  there  would  be  no 
significant  differences  on  the  pretest  totals  of  the  two 
groups.   The  results  of  the  analysis  of  scores  are  presented 
in  Table  6. 

The  computed  t-value  was  compared  with  a  distribution 
of  t-tables  from  the  Students  t-table.   The  values  found 
indicated  that  for  50  degrees  of  freedom,  the  t-value  of  1.74 
at  the  .05  level  of  significance  should  be  used.   [Ref.  13] 

The  computed  value  of  t  for  null  hypothesis  one  has  led 
to  its  rejection  and  acceptance  of  the  alternate.   Therefore, 
there  was  no  significant  difference  between  the  pretest  scores 
of  the  Control  Group  when  compared  with  those  of  the.  Course 
Group.  •  Therefore,  comparing  the  performance  of  the  two 
groups  on  the  post  test  to  detect  any  improvement  by  the  Course 
Group  is  acceptable. 

Hypothesis  two  states  that  there  would  be  no  equivalency 
between  the  post  test  scores  of  the  Control  Group  and  those 
of  the  Course  Group,  with  the  scores  of  the  Course  Group  being 
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greater  than  those  of  the  Control  Group  by  at  least  one 
standard  deviation.   This  hypothesis  assumes  equivalency  in 
the  scores  on  the  pretest  by  both  groups. 

When  all  participants  had  completed  the  post  tests,  the 
combined  scores  for  both  goups  were  compared.   The  mean  post 
test  score  for  the  Control  Group  members  was  49.70  with  a 
standard  deviation  of  16.48 ,   The  mean  post  test  score  for 
the  Course  Group  students  was  93.39  with  a  standard  devia- 
tion of  25.14.   This  analysis  (Table  7)  resulted  in  a  sig- 
nificantly high  t-value.   The  difference  between  the  Control 
Group  and  Course  Group  was  demonstrated  and  therefore  the 
null  hypothesis  two  was  rejected. 

The  post  test  results  derived  from  cummulative  scores  of 
all  participants  (Appendix  G)  are  provided  for  further  study 
and  test  development.   A  total  of  176  points  was  possible  on 
the  post  test  which  covered  content  in  the  areas  of  (1)  task 
analysis,  (2)  selection  of  personnel  for  testing,  (3)  work- 
space design  and  arrangement,  (4)  control  panel  design, 
(5)  individual  information  processing  capabilities,  (6)  envi- 
ronmental factors  which  may  affect  human  performance  -  units 
of  measure,  (7)  environmental  considerations  inclusive  in 
design,  (8)  objectives  of  taking  a  human  factors  course,  and 
(9)  factors  contributing  to  human  error. 

A  summary  of  these  results  for  the  Course  Group  (Table  8) 
shows  that  all  content  area  scores  were  at  least  50  per  cent 
of  the  total  possible  points  with  the  exception  of  "Control 
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Panel  Design",  "Individual  Processing  Capabilities",  and  "En- 
vironmental Factors  Affecting  Human  Performance"  which 
were  rated  as  46.38  and  48  per  cents  respectively. 

A  summary  of  these  results  for  the  Control  Group  (Table  8) 
shows  that  only  one  content  area  (Factors  Which  Cause  Human 
Error)  scored  at  least  50  per  cent.   All  others  scored  less 
than  50  per  cent  with  a  range  of  3-48  per  cent. 

Additionally,  the  test  questions  were  redistributed  under 
each  of  the  enabling  objectives  (Table  2)  which  they  support- 
ed: 

1.  Given  a  list  of  human  factors  terms,  write  a  defini- 
tion for  each  in  your  own  words. 

2.  Given  a  list  of  specific  HFE  problems  and/or  areas 
of  concerns,  identify  which  human  factors  references  should 
be  consulted  to  provide  solutions  and  guidelines. 

3.  Given  at  least  one  control/display  example,  use  the 
appropriate  human  factors  reference  to  determine  which  features 
need  to  be  redesigned. 

5,  Explain  in  your  own  words  the  importance  of  evaluat- 
ing the  human  factors  aspects  of  systems  and  equipments. 

6.  Identify  the  human  factors  principles  which  must  be 
applied  during  the  evaluation  of  a  workspace  and  control/ 
display  panel. 

9.   Given  a  list  of  10  factors  and  forces  which  affect 
human  performance,  identify  what  aspect  of  human  performance 
each  affects . 
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10.   Given  a  list  of  10  factors  and  forces  which  affect 
human  performance,  identify  what  unit  of  measure  must  be  used 
to  assess  the  amount  of  protection  required  to  diminish  their 
adverse  effects  upon  the  human  operator. 

All  the  scores  for  the  Course  Group  (Table  9)  rated  at 
lease  50  per  cent  with  the  exception  of  Objective  6  which 
rated  45  per  cent  and  Objective  9  which  rated  49  per  cent. 

The  Control  Group  scores  (Table  9)  failed  to  reach  50 
per  cent  with  the  exception  of  Objective  6  which  rated  51 
per  cent. 

Since  the  test  questions  had  received  only  minimal  pre- 
testing, the  results  warrant  further  test  item  development 
and  evaluation.   Although  no  specific  conclusions  will  be 
reported  based  on  this  preliminary  analysis,  it  can  be 
noted  that  the  Course  Group  obtained  higher  scores  than  did 
the  Control  Group  in  all  but  one  of  the  content  areas  and  in 
all  of  the  enabling  objectives.   (See  right  columns  of  Tables 
8  and  9). 

C.   QUESTIONNAIRE  ANALYSIS   .  " 

Three  separate  questionnaires  were  designed  and  adminis- 
tered in  accordance  with  NAVEDTRAs  106A,  llOA  and  the  Ques- 
tionnaire Construction  Manual  developed  for  the  Ft.  Hood  Field 
Unit  by  the  U.  S.  Army  Research  Institute  for  the  Behavioral 
Social  Science.   Summary  information  on  the  design  and  use 
of  each  questionnaire  is  provided  in  Table  10.   Every  effort 
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was  made  to  include  questions  on  the  rating  forms  that  the 
specific  users  were  best  qualified  to  answer. 

1.   Student  Attitude  Survey  (Questionnaires) 

Students  who  had  been  able  to  complete  at  least  five 
lessons  were  required  to  complete  the  Student  Questionnaire 
on  various  aspects  of  the  self-paced  instruction  received. 
There  were  24  students  in  this  classification.   Students  were 
directed  to  answer  only  those  questions  that  were  applicable 
to  the  material  they  completed.   Therefore,  the  reader  will 
note  that  the  total  number  of  responses  for  each  question  may 
tend  to  vary. 

Rating  scales  employed  in  the  student  questionnaires 
were  based  on  a  balanced  five-point  Likert  Scale.   The  num- 
ber of  students  selecting  each  descriptor  was  tabulated. 
Ordinal  values  of  1  to  5  were  given  to  student  ratings,  with 
5  being  the  most  positive  and  1  being  most  negative.   These 
ratings  were  then  divided  into  three  groups  and  interpreted 
as  "positive  (5-4),  "neutral"  (3),  and  "negative"  (2-1). 
The  frequency  of  students  responding  to  the  two  most  positive 
responses  was  combined  as  was  the  frequency  of  responses  for 
the  two  most  negative.   These  figures  were  then  translated 
into  a  percentage  of  the  total  number  of  students  responding. 
The  mean  and  standard  deviation  of  the  students'  responses 
for  each  question  were  also  calculated  using  the  formulas 
listed  below: 
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n 

£ 


T  =  E     '       ' 


X-^     X. 


i  =  l 


n  -  1 


The  mean  represented  the  degree  of  positiveness  of  the  Course 
Group's  responses  to  each  question.   These  responses  were 
used  to  measure  their  attitudes  and  assist  in  the  assessment 
of  the  various  aspects  of  the  course  and  administrative  pro- 
cedures such  as : 

a.  Course  Content  and  Objectives 

b.  Course  Design 

c.  Course  Presentation 

d.  Student  Supplement 

e.  Resource  Documents 

f.  Human  Factors  Attitudes 

g.  Miscellaneous 

2 ,   Command  Course  Supervisor  (CCS)  Questionnaire  and  Log 
Once  CCS  was  selected  from  each  of  the  four  partici- 
pating commands.   The  CCSs  maintained  a  log  throughout  the 
evaluation  period  and  upon  its  completion  filled  out  a 
questionnaire. 
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Rating  scales  employed  in  these  questionnaires  were 
based  on  a  numerical  cell  interval  freq-uency  scale.   The  fre 
quency  of  responses  was  grouped  into  three  cells: 

0  -   401 
41  -   60% 


61  -  100 


The  CCSs  estimated  the  frequency  of  student  questions  and 
comments  that  dealt  with  specific  areas  for  each  of  several 
pertinent  sections.   The  purpose  of  the  data  collected  was 
to  verify  and  clarify  student  questions,  comments  and  sub- 
stantiate the  use  of  the  course  materials  without  the  aid  of 
an  on-site  instructor.   CCSs  were  also  directed  to  assess 
student  attitudes  and  behaviors  during  the  evaluation 
based  upon  their  observations,  student  interactions  and  log 
entries.   These  responses  were  used  to  assist  in  the  assess- 
ment of  the  following  areas: 

a.  Lesson  Book  Materials 

b.  Student  Supplement 

c.  Resource  Documents 

d.  Motivation  and  Attitudes 

e.  Student  Information 

f.  Additional  Comments 

3.   Curriculum  Development  Exf^ert   Questionnaires 

Three  proven  specialists  in  the  field  of  curriculum 
evaluation  who  regularly  apply  the  standards  and  criteria 
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provided  by  NAVEDTRAs  106A  and  llOA  were  asked  to  assess  the 
extent  that  each  o£  the  terminal  objectives  satisfied  the 
six  criteria  identified  in  Appendix  E.   The  average  number 
of  years  of  experience  for  these  experts  was  5.5  years.   All 
were  civilians  in  government  service  whose  full  time  job  was 
to  evaluate  various  aspects  of  curriculum  prior  to  its  vali- 
dation and  approval  for  use  in  the  instructional  systems  of 
Navy  schools. 

Rating  scales  employed  in  this  questionnaire  were 
based  on  a  balanced  four-point  Likert  Scale,   No  neutral 
response  was  provided.   This  scale  measured  the  degree  to 
which  each  terminal  objective  met  each  of  the  six  criteria 
listed  in  Appendix  E.   Ordinal  values  ranging  from  1  through 
4,  were  applied  to  the  verbal  descriptors  as  follows: 

4  3  2  1 

/ L L / 

Completely   Mostly     To  Some    Not  At 

Degree       All 

D.   INTERNAL  REVIEW  AND  QUESTIONNAIRE  RESULTS 

The  results  for  the  Internal  Review  summarized  in 
Chapter  IV  and  specifically  discussed  in  Appendix  E  are  based 
on  questionnaire  responses  from  students  and  curriculum  de- 
velopment expert,>  as  well  as  this  researcher's  personal  eval- 
uation. 

Hypothesis  three  assumes  that  the  Course  Terminal  Objec- 
tives, Design,  Content,  and  Presentation  would  not  fail  to 
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meet  the  standards  and  criteria  in  accordance  with  NAVEDTRAs 
llOA  and  106A. 

Hypothesis  four  assumes  that  the  course  materials  would 
not  adversely  affect  student  attitude  regarding  the  need  for 
human  factors  engineering  test  evaluation. 

As  is  evidenced  in  the  results  which  follow,  the  alter- 
nate hypothesis  three  for  Course  Design  is  rejected  with  the 
failure  of  the  Learning  Objectives  to  follow  a  logical  pro- 
gressive sequence  which  is  supportive  of  the  order  of  the 
content  and  lesson  topics  in  Module  1.   The  layout  and  format 
are  deficient  in  the  following  areas:   (1)  information  prior 
to  the  regular  progress  checks  at  the  end  of  the  "information 
blocks",  (2)  remediation  for  selection  of  "wrong  answers", 
(3)  professionalism  in  the  quality  of  the  story  line,  (4)  in-  - 
formation  in  the  student  supplement  to  allow  its  independent 
usage  as  an  HFE  test  and  evaluation  planning  guide,  and 
(5)  ease  and  usability  of  the  course  lesson  books  caused  by  the 
"page  flipping"  aspect  of  the  course  procedures. 

The  null  hypothesis  three  for  Course  Content,  Presentation., 
and  Design  (sequencing  of  knowledge  and  skills  factors  con- 
tained in  Module  one)  is  rejected  in  as  much  as  these  areas 
satisfied  the  criteria  stated  in  Appendix  E.   However,  while 
the  lesson  topics  in  Modules  2  and  3  are  not  officially  part 
of  this  review,  it  must  be  noted  that  the  content  failed  to 
provide  prerequisite  knowledge  and  skills  required  for  under- 
standing and  demonstrating  follow-on  concepts  and  skills. 
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The  alternate  hypothesis  four  was  rejected  in  a  much  as 
92  per  cent  of  the  students  upon  course  completion,  responded 
that  HFE  test  and  evaluation  was  "extremely  important"  (Ap- 
pendix H) .   The  students'  attitudes  toward  human  factors 
testing  were  not  adversely  affected, 

1.   Course  Learning  Objectives  (LPs) 
a.   Job  Task  Analysis  (JTA) 

No  approved  JTA  was  used  to  develop  or  evaluate 
the  LOs ,  as  none  was  available.   The  sponsoring  military 
commands  developed  a  list  of  task  requirements  (Appendix  B) 
obtained  from  interaction  with  human  factors  engineers.   This 
list  was  provided  to  the  course  designers  to  guide  the 
development  of  the  course.   This  procedure  is  acceptable 
when  no  JTA  exists. 

Students  assess  the  importance  of  the  LOs  by 
identifying  those  which  they  considered  necessary  in  the 
planning  and  designing  of  HFE  test  and  evaluation.   The 
results  are  summarized  in  Table  11.   With  the  exception  of 
LOs  3,  4,  8,  10,  14,  and  17  all  the  remaining  LOs  were  judged 
by  at  least  40  per  cent  of  the  students  to  be  important  in 
planning  and  designing  for  HFE  test  and  evaluation. 

Students  also  assess  the  importance  of  the  LOs 
by  identifying  those  which  they  considered  necessary  in  the 
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performance  of  their  job,  which  for  most  was  general  test- 
and  evaluation.    [Ref.  3:  pp.  36-38,  79-80]   The  results 
are'  summarized  in  Table  12.   With  the  exception  of  LOs  8,  10, 
11,  12,  and  17  at  least  40  per  cent  of  the  students  judged 
the  remaining  LOs  as  important, 
b.   LO  Criteria 

The  Curriculum  Development  Experts  (CDEs)  were 
given  a  questionnaire  which  provided  a  list  of  course  LOs  and 
asked  to  rate  the  extent  to  which  each  LO  met  the  six  previ- 
ously specified  criteria  (Table  13) .   According  to  the  results 
of  their  evaluation,  no  objective  met  the  criteria  "Mostly" 
or  "Completely".   It  would  appear  that  the  LOs  did  not 
(1)  state  the  objective  in  terms  of  the  learner,  (2)  state 
standards,  and  (3)  state  the  conditions  under  which  learning 
is  to  occur.   Most  of  the  LOs  were  not  behaviorally  specific. 
Raw  data  is  provided  in  Appendix  I.     . 

The  alternate  hypothesis  three  is  therefore  re- 
jected with  the  failure  of  the  course  terminal  objectives 
to  meet  the  criteria  identified  in  NAVEDTRA  106A  and  the 
absense  of  module  (unit/section)  and  lesson  enabling  objec- 
tives. 

2 .   Course  Design 

a.   Sequence  and  Structure 

Of  primary  importance  in  support  of  the  "building 
block"  concept  of  course  development  is  the  logical  progres- 
sion of  knowledge  and  skill  factors  addressed  by  the  terminal 
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objectives  and  the  course  content.  This  also  assumes  that 
the  terminal  objectives- will  be ' presented  in  the  same  pro- 
gressive order  as  the  lesson  topics. 

To  conduct  this  evaluation,  the  intent  of  purpose 
of  each  learning  objective  was  identified  and  broken  down  by 
this  researcher  into  four  areas:   (1)  knowledge  of  concepts, 
(2)  knowledge  of  skills,  (3)  application  of  knowledge  and 
(4)  application  of  skills.   The  HFE  course  materials  were 
then  reviewed,  identifying  lessons  which  satisfied  the  in- 
tent or  purpose  of  each  objective  (Table  14).   NAVEDTRAs  llOA 
and  llOA  require  that  a  student  be  tasked  to  perform  a  skill 
only  after  the  prerequisite  knowledge  factors  have  been  pre- 
sented.  This  breakdown,  therefore,  enabled  this  researcher 
to  evaluate  and  compare  the  logical  progression  of  the  termi- 
nal objectives  and  the  lesson  topics.   Additionally,  student 
attitude  survey  responses  were  assessed. 

In  this  researcher's  opinion,  all  20  lesson 
topics  within  the  first  module  appear  to  be  supportive  of 
the  "building  block"  concept  for  development  of  knowledge 
and  skills.   Fifty  eight  and  66  per  cent  of  stud'ent  responses 
(Table  15)   to  questions  Bl  and  B2  respectively  are  supportive 
of  this  opinion.   In  completing  lessons  beyond  the  first 
module,  however,  some  discontinuity  appears,  i.e.,  statisti- 
cal concepts  (lessons  33  and  34)  such  as  probability  of 
success,  confidence  levels  and  probability  of  failure-free 
system  operation  are  addressed  in  lessons  22  and  26 
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respectively.   Additionally,  technical  references  (.lessons 
36  and  37)  such  as  HFTEMAN,  HEDGE  and  MOAT  are  introduced 
in  lessons  23  and  24.   Students  are  told  that  more  detailed 
information  regarding  these  documents  will  be  presented 
later.   This  researcher  needed  more  information  about  these 
documents  to  clarify  points  being  made  in  lessons  23  and  24 
and  was  required  to  search  for  the  information  in  lessons  36 
and  37.   It,  therefore,  becomes  apparent  to  this  researcher 
that  modules  two  and  three  need  to  be  reviewed  to  ensure 
that  a  logical  progression  of  the  knowledge  and  skills 
factors  is  provided. 

In  this  researcher's  opinion,  the  terminal  ob- 
jectives do  not  follow  a  logical  progression  as  exhibited  by 
Table  14.   Specifically,  LO  3  which  required  students' 
knowledge  of  HFE  technical  documents  as  presented  in  lessons 
36  and  37,  is  sequenced  before  LO  4  which  addresses  HFE  goals 
as  presented  in  lesson  3.   Therefore,  LO  3  is  out  of  sequence 
or  the  material  in  lessons  36  and  37  should  be  presented 
earlier  in  the  sequence  (prior  to  lessons  23  and  24).   Table 
14  presents  other  examples  of  incongruencies  of  lesson 
materials.   Additionally,  at  least  50  per  cent  of  the  students 
responding  to  question  Al  (Table  16)  felt  that  objectives  1, 
2,  4,  and  7  were  addressed  within  the  first  20  lessons. 

Since  the  first  20  lessons  are  judged  to  be  logi- 
cally progressive  and  the  LOs  are  not,  they  therefore  do  not 
coincide  in  a  supportive  relationship.   As  additionally 
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required  by  NAVEDTRA  106A,  each  objective  must  be  taught  with- 
in the  course  of  instruction.   LO  11  required  the  student  to 
"Calculate  human  performance  reliability"  but  the  course 
does  not  teach  the  student  how  to  do  so  and  it  was  not  one 
of  the  pre-course  requirements. 

b.  Figures  and  Tables 

All  figures  and  tables  used  in  the  Student  Sup- 
plement were  rated  by  at  least  65  per  cent  of  the  student 
Course  Group  as  "positive"  (Table  17).   Written  comments 
were  received  from  participants  which  stated  that  while 
figures  and  tables  were  good,  without  the  lesson  books  to 
provide  additonal  explanation,  many  of  them  could  not  be 
fully  understood. 

c.  Symbols  and  Legends 

All  symbology  and  legends  used  in  the  Student  . 
Supplement  were  rated  by  at  least  63  per  cent  of  the  Course 
Group  as  "effective"  (Table  18) .   No  additonal  comments  were 
received  from  students. 

d.  Layout  and  Format 

Results  of  student  attitudes  regarding  the 
effectiveness  of  the  layout  (Student  Supplement,  eight  lesson 
books,  and  resource  documents)  are  displayed  in  Table  19. 
Comments  indicated  that  it  did  not  hinder  their  learning 
experience  and  many  indicated  they  preferred  the  way  the 
lesson  topics  were  divided  into  separate  lesson  books.   They 
remarked  that  this  added  to  the  portability  of  the  course 
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materials.   Addi tonally,  the  students  felt  that  much  of  the 
material  in  the  Student  Supplement  had  to  be  clarified  by 
information  in  the  lesson  books.   It  would  therefore,  be 
very  difficult  to  use  the  Student  Supplement  as  a  stand-alone 
reference  and  planning  guide  for  HFE  test  and  evaluation. 
They  considered  its  value  as  negligable  once  the  student 
lesson  books  were  returned  to  the  sponsoring  agency  upon 
completion  of  the  course. 

Regarding  the  format,  no  less  than  50  per  cent 
responded  positively  to   (1)  Course  Procedures,  (2)  Informa- 
tion Blocks,  and  (3)  Quality  of  Questions.   It  must  be  noted 
that  both  course  procedures  and  quality  of  story  line  re- 
ceived a  negative  rating  by  42  per  cent  of  the  students. 
Written  and  verbal  comments  from  participants  indicated  a 
strong  dissatisfaction  with  "page  flipping"  aspect  of  the 
course  procedures  to  search  for  answers  as  well  as  an  equally 
strong  dislike  for  the  story  line  following  the  antics  of  the 
ficticious  Lt.  I.  M.  Eager. 

Addtional  amplifying  information  regarding  layout 
and  format  were  extracted  from  written  comments  on  the  ques- 
tionnaires and  verbal  comments  recorded  by  the  CCSs.   These 
have  been  paraphrased  or  quoted  directly  when  found  to  be 
representative  of  the  majority  of  the  course  participants. 

(1)   As  a  group,  course  participants  felt  that 
the  information  frames  sometimes  did  not  provide  all  the  in- 
formation necessary  to  answer  correctly  the  questions  at  the 
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end  of  the  information  frame.   In  fact,  comments  reported  that 
sometimes  the  information  was  not  given  until  the  following 
frame . 

(2)  The  reinforcement  provided  in  the  "wrong 
answer  frames"  was  judged  to  be  borderline.   Students  also 
commented  that  the  wrong  answer  replies  were  demeaning,  i.e., 
"Surely,  you  didn't  answer  that  way".   "A  simple  'correct' 

or  'incorrect'  with  some  amplifying  information  will  suffice." 
was  a  comment  which  was  representative  of  a  majority  of  the 
students. 

(3)  The  quality  of  the  questions  asked  at  the 
end  of  the  "information  frames"  was  considered  borderline. 
Comments  from  students  indicated  that  sometimes  the  questions 
could  be  answered  by  elimination  of  "ridiculous  answer  choices" 
while  others  had  to  be  guessed.-  Some  participants  felt 

that  the  questions  were  unclear  or  worded  poorly  so  as  to 
cause  confusion. 

(4)  Several  students  responded  that  problems 
such  as  those  already  mentioned  as  well  as  errors  in  the 
course  materials  caused  endless  loops  resulting  in  irritation 
and  loss  of  motivation  for  continuing  with  the  course.   As 
one  student  stated  "I  would  have  quit  by  lesson  five,  if  I 
hadn't  been  assigned  to  complete  the  course." 

(5)  While  student  questionnaires  indicated  the 
format  was  neutral,  fifteen  of  the  24  students  polled  chose 
to  write  comments  concerning  their  dissatisfaction  with  the 
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"page  flipping"  format.   They  reported  losing  their  places 
and  having  to  begin  again  from  frame  one  of  the  chapter  being 
studied.   This  format  may  have  prevented  looking  ahead,  but 
it  most  certainly  "prevented  review  of  any  materials".   Judg- 
ing from  the  forcefullness  and  emotion  emanating  from  actual 
comments,  the  following  quote  sums  it  up,  "In  all  the  service 
schools  and  courses  which  I  have  attended  and  taken  in  the 
past  years,  programmed  texts  were  an  integral  part  of  the 
curriculum.   I  have  never  seen  a  programmed  text  structured 
such  as  this  one.   In  all  honesty,  after  the  first  25  pages, 
my  interest  level  dropped  to  zero  and  remained  there.   Thumb- 
ing through  numerous  pages  to  find  correct  answers,  or  incor- 
rect answers  and  then  constantly  repeating  the  procedure... 
created  a  frustrating  situation  that  stifled  any  learning. 
This  page  shuffling  drill  detracted  from  any  tangible  bene- 
fits that  might  have  been  derived  from  the  material  presented." 

(6)  The  inclusion  of  a  story  line  following  the 
adventures  of  the  ficticious  Lt.  I.  M.  Eager  and  his  buddy- 
boss,  Capt.  B.  Smart  was  not  accepted  well  by  course  parti- 
cipants. Of  all  the  questions  answered,  the  effectiveness 
of  the  story  line  was  rated  the  lowest.  While  the  numerical 
value  of  the  mean  places  it  at  the  "neutral"  point,  it  does 
so  by  only  .04  of  a  point  away  from  "negative". 

(7)   Fourteen  of  the  students  reported  that  the 
story  sequence  was  sometimes  disjointed  and  very  much  dis- 
tracting from  learning  the  essence  of  the  lessons.   Overall, 
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they  indicated  a  desire  to  "drop  the  junk  and  put  more  facts" 
"and  real  life  examples  in  its  place."   "The  stories  made 
the  text  too  'cute',  don't  try  to  entertain  me  in  a  course, 
try  to  teach  me.   The  way  the  material  was  presented  (using 
Eager)  just  irritated  me  and  the  effectiveness  of  the  course 
was  diminished," 

3.   Course  Content 

In  addition  to  the  overall  effectiveness  assessment 
provided  from  the  diagnostic  test  results,  an  important  indi- 
cator of  the  perceived  value  of  the  course  content  was  the 
number  of  students  desiring  to  keep  copies  of  the  Student 
Supplement  and  the  lesson  books  (Table  20).   Forty-two  per 
cent  of  the  course  participants  desired  to  keep  a  copy  of 
the  course  materials  and  54  per  cent  wanted  to  keep  the 
Student  Supplement. 

Table  20 

Student  Attitudes  Regarding  Retaining  a 
Copy  of  the  HFE  Course  Materials  for  Future  Use 


Percentage  of  Students 
Responses  Responding 


Yes     No 


Want  to  keep  course  materials?      42%      58% 
Want  to  keep  supplement  only?       54       46 
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4 .   Course  Presentation 

The  percentage  of  student  attitudes  regarding  the 
effectiveness  of  the  course  length  (Table  21)  was  calculated 
with  52  per  cent  responding  positively  and  26  per  cent  neg- 
atively.  Percentages  of  those  regarding  terminology  used 
were  65  per  cent  positive  and  9  per  cent  negative.   Responses 
regarding  participation  in  the  course  without  the  aid  of  an 
on-site  instructor  were  relatively  evenly  distributed  with 
37  per  cent  positive  and  34  per  cent  negative. 

Regarding  the  latter  set  of  responses,  it  is  believed 
that  contacting  the  CCS  (who  was  not  HFE  trained)  for  ex- 
changing books  and  receiving  tests  and  questionnaires  may 
have  been  interpreted  by  students  as  contact  with  an  on-site 
instructor.   The  CCSs'  logs  indicated  that  students  seldomly, 
if  ever,  required  their  assistance  with  course  content  in 
the  role  of  an  on-site  instructor.   Additionally,  this  re- 
searcher received  no  communication  from  students  asking  for 
clarity  of  course  instructions  or  content,  even  though  it 
was  encouraged  in  students'  instructions.-. 

It  must  also  be  pointed  out  that  42  per  cent  of  the 
course  participants  desired  to  keep  a  copy  of  the  course  les- 
son materials,  even  though  they  had  been  informed  that  no 
assistance  would  be  available  upon  completion  of  the  evaluation 
period. 

Additional  amplifying  information  regarding  course 
presentation  was  extracted  from  written  comments  on  the 
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questionnaires  and  verbal  comments  recorded  by  the  CCSs. 
These  have  been  paraphrased  or  quoted  directly  when  repre- 
sentative of  the  majority  of  the  course  participants: 

a.  The  level  of  detail  presented  throughout  the 
course  materials  presented  no  difficulty  in  content  compre- 
hension according  to  student  reponses  in  the  questionnaires. 
Some  comments  were  made  which  indicated  that  students  would 
like  to  see  an  even  greater  detail  for  some  areas,  however, 
they  were  not  specific  as  to  which  areas. 

b.  No  negative  comments  were  received  regarding  the 
course  length.   The  time  required  for  course  participation 
was  not  overburdening  to  them.   However,  students  felt  that 
the  format  caused  time  to  be  wasted  which  could  have  been 
spent  on  the  course  content. 

c.  The  only  feedback  students  received  while  taking 
the  HFE  course  was  provided  in  the  answers  at  the  end  of  each 
information  frame.   No  periodic  self-check  tests  were  avail- 
able with  the  course  materials .   According  to  the  course 
sponsors,  none  have  as  yet  been  developed.   The  sponsors  had 
intended  that  measurement  of  the  students'  attainment  of  the 
course  LOs  would  not  be  necessary  as  their  mastery  of  the 
content  would  become  apparent  as  it  was  applied  in  the  normal 
performance  of  their  jobs  in  designing  and  implementing  tests 
for  evaluation  of  the  human  factors  aspects  of  equipments  and 
systems . 
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5.  Course  Availability 

According  to  the  course  sponsors,  original  plans 
called  for  the  U.  S.  Army  to  supply  all  necessary  materials 
to  its  own  personnel,  while  the  cognizant  Navy  commands 
would  fulfill  Navy  requests,  once  the  course  was  finalized. 
However,  present  constraints  on  budget  and  personnel  to  make 
corrections  and  implement  the  computer  program  may  hinder, 
if  not  prevent,  this  effort. 

6 .  Course  Maintainability 

Future  plans  of  perceived  needs  in  this  area  cannot 
be  reported,  as  none  were  received  from  the  sponsoring  com- 
mands.  However,  even  during  this  preliminary  evaluation,  its 
importance  was -clearly  noted.   At  present,  course  materials 
refer  to  specific  pages  of  MIL-STDs  and  handbooks  which 
caused  some  confusion  to  course  participants  when  using  MIL- 
STD  1472C.   The  HFE  course  makes  references  to  specific 
pages  within  MIL-STD  1472B.   In  the  latest  version  of  the  MIL- 
STD,  the  pages  had  been  renumbered  and  students  stated  that 
there  was  some  difficulty  in  finding  the  correct  figure  or 
information  they  were  assigned  to  read.   Additionally,  the 
United  States  has  converted  their  measuring  system  to  metrics 
and  this  is  not  reflected  within  the  course  materials. 

7 .  Course  Supportability 

Future  plans  of  perceived  needs  in  this  area  cannot 
be  reported  as  none  were  provided  from  the  sponsoring  commands 
At  present,  supporting  commands  remain  to  be  identified. 
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V.   CONCLUSIONS  AND  RE  COMMEND  A.T  IONS 

A.  CONCLUSIONS 

The  Human  Factors  Engineering  Self-Paced  Course  has  the 
potential  to  be  most  effective,  based  on  its  demonstrated 
ability  to  successfully  provide  the  stimuli  for  the  transfer 
of  knowledge  and  skills  to  the  Course  Group  in  support  of 
the  intent  of  the  course  originators'  task  requirements  and 
terminal  objectives.   The  incorporation  of  the  recommendations 
provided  below  would  provide  the  course  the  potential  to  pass 
a  validation  attempt  in  accordance  with  CNTECHTRA  requirements. 

Further  validation  of  course  content  validity  should  in- 
clude a  study  of  the  Course  Group  participants  after  having 
had  the  opportunity  to  apply  their  newly  gained  knowledge 
and  skills  in  the  performance  of  their  jobs  in  designing  and 
conducting  human  factors  tests. 

B.  RECOMMENDATIONS 

It  is  recommended  that  the  course  designers/sponsors 
comply  with  the  standards  and  criteria  identified  by  CNTECHTRA 
in  NAVEDTRAs  llOA  and  106A  for  the  development  of  military 
educational  courses  and  incorporate  the  following  changes  into 
the  Human  Factors  Engineering  Self -Paced  Course. 
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1.  Course  Learning  Objectives 

Rewrite  and  rearrange  course' terminal  objectives  and 
develop  lesson  and  module  enabling  objectives.   Incorporate 
the  elements  required  for  acceptable  learning  objectives  which 
appear  to  be  missing  from  those  presently  given  for  this 
course:   (1)  specific  description  of  expected  learner .action, 

(2)  conditions  under  which  the  action  will  take  place,  and 

(3)  standards  or  criterion  which  must  be  reached  by  the  stu- 
dents.  Evaluate  them  again  according  to  the  six  criteria  re- 
quired by  NAVEDTRA  106A.   The  17  course  terminal  learning 
objectives  have  been  restructured  (Table  22)  to  meet  the  cri- 
teria.  The  learning  objectives  should,  however,  be  presented 
in  the  same  order  as  the  course  content.   A  recommended  re- 
ordering of  the  course's  original  terminal  objectives  have 
been  provided  in  Table  23. 

2.  Course  Design 

a.  Sequence  of  Course  Content. 

Reevaluate  the  sequence  of  the  content  as  it  is 
presently  given  to  ensure  that  knowledge  and  concepts  are 
taught  before  the  students  are  required  to  apply  them,  i.e., 
task  analysis  before  human  performance  evaluation.   This  will 
support  the  "building  block"  concept  of  learning  skills  and 
provide  the  student  a  better  opportunity  to  attain  success 
and  avoid  unnecessary  frustrations. 

b.  Layout  of  Student  Supplement  and  Lesson  Books. 
Follow  the  guidelines  provided  in  NAVEDTRA  llOA 

which  addresses  the  proper  physical  layout  and  required 
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Table  22  . 

Examples  of  Recommended  Restructured  Terminal 
Learning  Objectives  lAW  NAVEDTRA  106A 

1.  Define  in  writing  common  terms  used  in  human  factors 
engineering  in  accordance  with  lessons  1-40. 

2.  Given  a  list  of  HFE  reference  documents,  select  the  major 
topics  addressed  by  each  in  accordance  with  lessons  1-40. 

3.  Given  an  HFE  problem,  demonstrate  the  ability  to  resolve 
it  by  correctly  selecting  the  HFE  reference  and  locating 
within  its  contents  the  section  which  provides  the  ne- 
cessary information  in  accordance  with  lessons  1-40. 

4.  Describe  in  writing  the  basic  goals  of  HFE  in  a  materiel 
development  program  in  accordance  with  lessons  1-4. 

5.  Describe  in  writing  the  inter-relationships  of  the  four 
major  factors  of  HFE  in  a  materiel  development  program 
i.e.,  human  performance  requirements,  personnel  selection 
criteria,  training  and  equipment  design- in  accordance 
with  lessons  1-4  and  28-29. 

6.  Given  the  stages  of  the  basic  systems  acquisition  cycle, 
describe  in  your  ovm  words  the  level  of  involvement  an 
HFE  designer  and  evaluator  should  have  at  each  point. 
Include  in  your  discussion  the  purpose  of  your  testing 
and  the  basic  steps  you  would  follow  to  become  involved 
at  each  stage.   Responses  should  be  in  accordance  with 
lessons  21  and  22. 

7.  Discuss  in  writing  the  importance  and  purpose  of  evalua-  ' 
ting  human  factors  in  a  systems -concept  rather  than  just " 
evaluating  the  operator  or  the  machine  individually  in 
accordance  with  lessons  2,  21,  and  22. 

8.  Given  an  HFE  proglem,  analyze  systems  concept  employing 
task  analysis  and/or  determine  the  human  factors  require- 
ments which  must  be  considered  when  designing  the  piece 
of  equipment  in  accordance  with  lessons  6,  7,  14,  and 
21-24. 

9.  Given  an  HFE  problem,  identify  3  factors  and  3  forces 
which  would  affect  human  performance  and  describe  how  you 
would  measure  each,  i.e.,  what  equipment  to  use  and  what 
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unit  of  measure  is  appropriate,  in  accordance  with  lessons 
9,  10,  and  15-19. 

10.  Discuss  in  writing  the  differences  between  HFE  measure- 
ments taken  in  the  field  and  those  later  in  a  laboratory 
setting.   Include  in  your  discussion  the  potential  dif- 
ficulties of  applying  generalized  laboratory  data  to 
field  situations,  in  accordance  with  lessons  31  and  32. 

11.  Discuss  in  writing  the  importance  of  performing  testing 
both  in  a  controlled  environment  and  that  in  the  "real 
world".   Responses  are  to  be  in  accordance  with  lessons 
31  and  32. 

12.  Given  a  list,  identify  the  "experimental  control"  measures 
necessary  in  any  test  involving  the  human  performance  and 
describe  the  probable  effects  on  the  test  data  if  they 
are  absent,  in  accordance  with  lessons  31  and  32. 

13.  Given  an  HFE  problem  with  human  performance  data  available 
calculate  system  performance  reliability  and  assess  the 
potential  effectiveness  of  the  system.   Round  your  answer 
to  the  nearest  thousandths.   Responses  are  to  be  in 
accordance  with  lessons  21,  22,  26,  33  and  34. 

14.  Given  three  different  examples  of  a  task  analysis  for  the 
same  piece  of  equipment,  select  the  one  which  is  done 
correctly,  in  accordance  with  lessons  23  and  24. 

15..  Given  an  HFE  problem,  determine  performance  measures  for 
the  dependent  variables  of  time  and  error  in  accordance 
with  lessons  33  and  34. 

16.  Describe  the  relationship  between  human  factors  engineer- 
ing and  the  engineering  specialties  of  reliability, 
maintainability  and  safety  in  accordance  with  lessons 
25-27. 

17.  Given  an  HFE  problem,  state  the  HFE  standards  and  speci- 
fications necessary  to  evaluate  human  performance  in  ac- 
cordance with  the  guidelines  and  reference  documents 
presented  in  lessons  21  and  22. 

18.  Interpret  the  data  given  in  an  analysis  of  variance  table 
performance  scores  from  an  experiment  having  three  inde- 
pendent variables  with  one  statistically  significant 
first  order  interaction,  in  accordance  with  lessons  33 
and  34. 


99 


Table  23 

Recommended  Resequencing  of  the  Human  Factors  Engineering 
Self -Paced  Course  Terminal  Learning  Objectives 

Upon  completion  of  the  Human  Factors  Engineering  Course,  the 
student  will  demonstrate  the  following  capabilities  and 
knpwledge: 

1.  An  understanding  of  common  terms  used  in  human  factors 
engineering. 

2.  A  familiarity  with  human  factors  references  and  an  ability 
to  use  them. 

3.  An  awareness  of  potential  sources  of  technical  information 
on  major  human  factors  topics. 

17.   An  ability  to  interpret  the  standards  and  specifications 
of  the  human  factors  engineering  community,  through  the 
use  of  HFE  references. 

4.  An  understanding  of  the  goals  of  human  factors  engineering 
in  a  materiel  development  program. 

5.  A  familiarity  with  the  acquisition  cycle. 

7.   An  understanding  of  the  kinds  of  factors  and  forces  which 
affect  human  performance  and  an  ability  to  identify  them. 

15.   A  familiarity  with  task  analysis. 

7.   An  ability  to  measure  the  factors  and  forces  which  affect 
human  performance. 

10;   An  understanding  of  basic  statistical  techniques,  such  as 
an  analysis  of  variance. 

12.   An  ability  to  formulate  performance  measures  for  the  de- 
pendent variables  of  time  and  error. 

11,   An  ability  to  calculate  human  performa-nce  reliability. 

5.   An  ability  to  integrate  human  factors  principles  in  a 
DOD  sponsored  program. 

14.   An  understanding  of  the  major  techniques  used  by  human 
factors  specialist  during  system  synthesis,  design,  and 
development . 
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6.   An  ability  to  determine  human  performance  requirements 
in  a  systems  concept. 

13.   An  ability  to  analyze  human  performance  data  within  the 

context  of  "system  effectiveness"  and  "system  reliability" 

16.  An  awareness  of  the  relationship  between  human  factors 
engineering  and  the  engineering  specialist  of  reliability, 
maintainability  and  safety. 

8.  An  awareness  of  the  differences  between  field  and  labora- 
tory measurements . 

9.  An  awareness  of  what  "experimental  control"  measures  are 
necessary  for  any  test  involving  human  performance  and 
the  effects  in  their  absence. 

17.  An  ability  to  apply  the  standards  and  specifications  of 
the  human  factors  engineering  community. 
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elements  of  module  (lesson)  booklets  and  lesson  topics: 

(1)  Instructional  materials  are  required  to  have 
a  specific  numbering  system  which  clearly  shows  the  relation- 
ships between  terminal  objectives,  enabling  objectives  and 
course  content. 

(2)  Each  module  booklet  is  required  to  have  a 
cover  page,  module  overview  and  lesson  topics.  Samples  of 
the  module  overview  and  cover  page  are  provided  in  Appendix  I. 

(3)  Lesson  topics  cover,  wholly  or  in  part, 
terminal  objectives.   Progress  checks  (self-tests)  are  pro- 
vided for  the  student  at  the  end  of  lesson  topics  to  measure 
their  ability  to  achieve  the  terminal  objectives  and  determine 
the  need  for  remediation. 

(4)  Each  lesson  topic  shall  contain  the  items 
listed  below: 

Lesson  Topic  Cover  Page* 
Lesson  Topic  Overview* 
List  of  Study  Resources 
Lesson  Topic  Summary 
Narrative  Form  of  Lesson  Topic 
Programmed  Instruction  Material 
Lesson  Topic  Progress  Check,  complete  with 
feedback  and  remediation. 

Samples  of  those  items  asterisk' d  above  are  provided  in 

Appendix  J. 
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c.   Format  of  Course  Lesson  Books 

(1)  Review  "information  frames"  and  questions  to 
ensure  that  sufficient  information  has  been  provided  to  answer 
the  questions  before  they  are  asked,  i.e.,  Lesson  15,  page  84, 
"frame  from  page  80";  Lesson  17,  page  68,  "frame  from  page  67" 

(2)  Review  "wrong  answer  frames",  increase  pro- 
fessionalism in  replies  and  provide  reinforcement  by  directing 
the  student  to  a  specific  area  to  help  choose  the  correct 
answer  on  the  next  try,  i.e..  Lesson  12,  page  68  "(3)   We 
know  you  don't  know,  but  we  want  you  to  give  it  a  shot.   Re- 
turn to  page  39...". 

(3)  Review  the  quality  of  the  question  asked. 
Rewrite  those  which  are  ambiguous,  require  guessing  or  really 
offer  no  doubt  as  to  the  answer  because  of  the  examples  pro- 
vided in  the  wording  of  the  answer  choices,   i.e..  Lesson  13, 
page  96,  "frame  from  page  93". 

(4)  Review  the  story  line  using  the  ficticious 
Lt,  I.  M.  Eager  and  if  it.  is  decided  to  maintain  the  story 
line,  upgrade  the  examples  and  remove  extraneous  portions 
which  do  not  add  to  the  content  of  the  course.   Provide  more 
"real  world"  examples  which  could  add  greater  meaning  and 
clarity  to  the  content. 

(5)  Change  the  "page  flipping"  format  of  the 
written  version  of  the  course.   The  U.  S.  Navy  correspondence 
courses  provide  an  acceptable  format  and  one  with  which  some 
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military  personnel  are  already  familiar.   Another  popular  for- 
mat is  used  in  the  Elementary  Algebra:  Lecture  Lab  by  Arthur 
H.  Heywood. 

3.  Course  Content. 

a.  Review  course  content  to  be  sure  that  the  tasks 
identified  in  the  learning  objectives  are  adequately  addressed, 
i.e.,  calculation  of  reliability. 

b.  Correct  clerical  errors,  i.e.,  Table  33.5  of 
Student  Supplement;  Lesson  20,  page  74,  says  "frame  from  page 
71"  when  it  should  read  "...page  24";  Lesson  19,  page  91,  frame 
from  page  26:  reads  "By  finding  the  point  of  ...  between  four 
hours...",  should  read  "...two  hours". 

4.  Course  Presentation. 

a.  Provide  student  self-check  tests  following  each 
lesson  topic  and  following  each  module  with  a  final  comprehen- 
sive test  upon  course  completion  (after  the  course  review). 
The  questions  at  the  end  of  each  information  frame  don't  allow 
the  student  to  adequately  evaluate  their  understanding  or 
acquisition  of  the  necessary  concepts  and  skills.   Include 
reinforcement  for  end-of -information-frame  tests  by  referring 
the  student  to  the  appropriate  sections  for  review  based  on 
students'  incorrect  answers. 

b.  Provide  course  tests  so  that  students  can  judge 
how  well  they  had  attained  the  course  learning  objectives  be- 
fore they  attempt  to  apply  it  in  their  normal  jobs.   This  will 
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allow  them  to  go  back  and  review  the  areas  in  which  they  are 
weak.   A  certificate  of  completion  and  credit  can  then  be 
awarded  upon  satisfactory  completion  of  the  course.   Comple- 
tion of  the  course  may  also  be  made  a  part  of  each  command's 
qualification  procedures. 

5 .   Course  Availability,  Maintainability,  and  Supportabil- 
ity. 

a.  Provide  plans  and  estimates  required  for  future 
implementation  of  the  HFE  Course  and  an  outline  of  the  ne- 
cessary s-upport  in  those  areas  as  specified  in  NAVEDTRA  llOA. 

b.  Update  this  version  of  the  course  to  be  consistent 
with  MIL-STD  1472C  vice  MIL-STD  1472B.   When  referencing 
material  within  any  of  the  reference  documents  used  within 
the  course,  do  not  use  page  numbers;  provide  section  numbers 
instead.   Addtionally,  make  references  more  specific  when 
dealing  with  MIL-STD  documents;  do  hot  just  refer  to  the 
document  as  a  whole  when  addressing  a  specific  section. 

c.  Recommend  a  schedule  of  periodic  course  review 
and  updates  as  indicated  by  NAVEDTRA  llOA. 
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VI.   SUMMARY 

As  this  study  indicates,  the  Human  Factors  Engineering 
Self-Paced  Course  has  the  strong  potential  for  providing 
the  stimuli  to  transfer  knowledge  and  skills  to  its  studfents, 
Whether  this  course  provides  sufficient  content  to  effective' 
ly  bridge  the  gap  between  the  military  technical  documents 
and  standards  to  allo;\r  for  their  increased  usage  by  evalu- 
ators  on  the  job  is  an  area  for  further  study. 

There  is  an  interesting  side  note  relative  to  the  low 
level  of  human  factors  familiarity  by  all  experiment  parti- 
cipants as  indicated  by  the  results  on  the  pretest:  could 
this  possibly  be  substantiation  to  the  claim  made  by  GAO  in 
their  report  to  the  U.  S.  Congress  [Ref.  1:  pp.  1-27]  which 
has  been  referenced  periodically  throughout  this  thesis? 
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APPENDIX  A. 
RESOURCE  DOCUMENTS  SUMMARIES 

A  brief  summary  of  the  various  military  standards,  hand- 
books, and  documents  required  for  use  by  the  Human  Factors 
Engineering  Self-Paced  Course  participants  is  provided 
within  this  Appendix: 

1.  MIL-H-46855B 

2.  MIL-STD-1472C 

3.  MIL-STD-1474B 

4.  MIL-HDBK-759 

5.  Army  Regulation  602-1 

6.  Technical  Memorandum  29-76 

Additonally,  summaries  or  copies  of  the  abstracts  are 
provided  for  documents  to  which  the  Human  Factors  Engineering 
Self-Paced  Course  referred,  but  are  not  essential  for  course 
completion: 

1.  MIL-STD-721B 

2.  FED-STD-595 

3.  Guide  to  Human  Factors  Engineering  General  Purpose 
Test  Planning  (GPTP) 

4 .  HFTEMAN 

5.  HEDGE 

6 .  MOAT 
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REQUIRED  DOCUMENTS   • 

1.   MIL-H-46855B  Human  Engineering  Requirements  for  Military 
Systems,  Equipments  and  Facilities 

'•MIL-H-46855B  establishes  and  defines  the  overall  require- 
ments for  applying  the  human  engineering  priciples  and  cri- 
teria presented  in  MIL-STD-1472C  during  the  procurement  of 
military  systems,  thereby  effectively  integrating  man  into 
the  system.   The  specification  requires  the  prospective  con- 
tractor to  state  his  approach  in  his  Human  Engineering  Pro- 
gram Plan  which  is  submitted  in  response  to  the  Request  for 
Proposal  (RFP) .   The  Human  Engineering  Program  Plan,  upon 
acceptance  of  hte  proposal,  becomes  part  of  the  procurement 
contract. 

The  three  major  areas  of  the  system  acquisition  process 
where  these  human  factors  engineering  principles  are  to  be 
employed  are  analysis,  design  and  development,  and  test  and 
evaluation.   The  objectives  of  analysis  are  identification 
and  definition  of  system  operations,  maintenance,  training 
and  control  functions;  allocation  of  these  functions  to  man 
and/or  machine;  analysis  of  the  tasks  comprising  these  func- 
tions; and  development  of  system  specific  human  engineering 
design  criteria  and  operation  and  maintenance  procedures. 
The  human  engineering  inputs  developed  in  the  analysis  phase 
and  comensurate  with  MIL-STD-1472C  are  to  be  incorporated 
into  the  detailed  design.   These  human  engineering  provisions 
shall  .be  evaluated  during  the  design  reviews  to  ensure  their 
adequacy.   The  purpose  of  the  test  and  evaluation  phase  is 
assurance  of  the  fulfillment  of  MIL-H-46855B  and  contract 
requirement,  demonstration  of  conformance  to  MIL-STD-1472C 
quantification  of  man-machine  system  performance,  and  indi- 
cation of  the  introduction  of  possible  undesirable  design  or 
procedural  features. 

■MIL-STD-1472C  and  MIL-H-46855B  are  complementary  direc- 
tives that  deal  with  requirements  for  implementation  of  human 
factors  engineering  principles  in  the  development  and  acqui- 
sition of  military  systems,  equipment  and  facilities.   The 
former  establishes  the  criteria  by  which  the  systems  are  to 
be  human  engineered,  and  the  latter  establishes  the  require- 
ments for  maintenance  and  maintainability  as  integral  parts 
of  the  human  engineering  of  the  total  system." 
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2,  MIL-STD-1472C.   Human  Engineering  Design  Criteria  for 
Military  Systems,  Equipments  and  Facilities. 

"This  standard  establishes  general  human  engineering  cri- 
teria for  development  of  military  systems,  subsystems,  equip- 
ment and  facilities. 

The  purpose  of  this  standard  is  to  present  human  engineering 
design  criteria,  principles,  and  practices  to  achieve  mission 
success  through  integration  of  the  human  into  the  system, 
subsystem,  equipment,  and  facility,  and  achieve  effectiveness, 
simplicity,  efficiency,  reliability,  and  safety  of  system 
operation,  training,  and  maintenance. 

More  specifically,  its  purpose  is  to  present  human  engineer- 
ing design,  criteria,  priciples  and  practices  to  be  applied  in 
the  design  of  systems,  equipment  and  facilities  so  as  to: 

a.  Achieve  required  performance  by  operator,  control  and 
maintenance  personnel. 

b.  Minimize  skill  and  personnel  requirements  and  training 
time. 

c.  Achieve  required  reliability  of  personnel -equipment 
combinations . 

d.  Foster  design  standardization  within  and  among  systems. 

The  standard  includes  a  compilation  of  anthropometric  data 
from  several  military  sources  (MIL-STD-721B,  MIL-STD-1474B 
and  FED-STD- 595) ,  an  extensive  collection  of  control/display 
design  criteria,  hazard  and  safety  considerations,  and  require- 
ments for  certain  specialized  systems  which  may  also  be  applied 
to  such  equipments  as  ground  vehicles,  remote  handling  devices, 
air  and  ship  crew  stations." 

3.  MIL-STD-1474B   Noise  Limits  for  Army  Materiel 

"Three  distinctly  different  types  of  'noise  criteria' 
which  are  used  to  limit  noise  exposure  have  evolved  over  the 
years.   It  is  important  to  distinguish  among  the  three  types 
in  order  that  the  proper  type  may  be  chosen  for  application  and 
use  in  various  situations.   The  three  types  of  noise  criteria 
are: 

a.  Hearing  damage-risk  criteria 

b.  Hearing  conservative  criteria 

c.  Materiel  design  standards 
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This  document  is  a  design  standard  for  noise.   It  is  based 
on  provisions  of  TB-MED-251  with  respect  to  noise  exposure 
criteria.  MIL-STD-1472C  extracts  communications  criteria  from 
it." 
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4.   MIL-HDBK-759.   Human  Factors  Engineering  Design  for  Army 
Materiel. 

"This- handbook  gives  the  design  engineer  both  human  factors 
engineering  design  priciples  and  detailed  criteria.   The  design 
principles  are  stated  as  general  rules  to  be  applied  during 
system-development  programs  or  .as  essential  items  that  must  be 
considered  during  design  to  insure  that- sound  human  factors 
engineering  practices  will  be  incorporated.   The  detailed  cri- 
teria consist  of  dimensions,  ranges,  tolerances,  and  other 
specific  data.   In  some  cases,  the  range  of  acceptable  dimen- 
sions and  other  factors  may  be  rather  large.   Where  only  the 
minimum  and  maximum  are  given,  design  engineers  may  select  any 
part  or  item  within  the  recommended  range.   But  where  optimum 
dimensions  are  given,  designers  should  aim  to  approximate  them 
whenever  possible. 

The  purpose  of  this  document  is  to  establish  in  handbook 
form  general  data  and  detailed  criteria  for  human  factors 
engineering  application  in  the  design  and  development  of  army 
materiel. 

The  information  in  this  handbook  is  a  consolidation  of 
the  material  contained  in  four  U.  S.  Army  Human  Engineering 
Laboratory  Standard  Documents;  HEL  S-2-64A,  S-3-65,  S-6-66, 
and  S-7-68..  The  user  will  also  note  the  similarities  of  this 
document  with  MIL-STD-1472C. " 


5.   Army  Regulation  602-1.   Personnel -Materiql  Systems :   Human 
Factors  Engineering  Program.   Washington,  D.  C,  June  1976 

"This  regulation  prescribes  policies  and  procedures  and 
assigns  .responsibilities  for  human  factors  engineering  (HFE) 
in  the  Department  of  the  Army.   For  the  purpose  of  this  reg- 
ulation, HFE  is  defined  as  a . comprehensive  technical  effort 
to  integrate  all  personnel  characteristics  (skills,  training, 
implications,  behavioral  reactions,  human  performance, 
anthropometric  data  and  biomedical  factors)  into  Army  doctrine 
and  systems  to  assure  operational  effectiveness,  safety,  and 
freedom  from  health  hazards." 


6.  TECHNICAL  MEMORANDUM  29-76.  Guide  for  Obtaining  and  An- 
alyzing Human  Performance  Data  in  a  Materiel  DevelopmelTt 
Proj  ect . 

"The  specific  objectives  of  this  report  are  to:   (1)  de- 
scribe how  to  conduct  and  report  an  HFE  test  according  to  the 
requirements  of  DI-H-1334A,  (2)  detail  the  expenditures  in 
time  and  money  associated  with  the  conduct  of  an  HFE  test, 
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(3)  provide  examples  of  HFE  test  reports  for  systems  in  "ex- 
perimental prototype"  and  "advanced  development"  phases  of 
development,  (4)  describe  the  uses  of  the  obtained  HFE  test 
data  as  a  function  of  system  development,  and  (5)  explain  the 
impact  of  the  DI-H-1334A  findings  on  a  materiel  development 
program. 

This  document  is  written  for  government  contract  monitors, 
contract  project  directors,  and  contractor  HFE  personnel. 
The  guidelines  for  conducting  and  reporting  on  the  HFE  test  'are 
intended  for  experienced  HFE  personnel. 

Questions  of  what  data  are  to  be  collected,  how  they  are  to 
be  collected,  and  how  the  data  can  be  used  are  discussed  in 
Chapters  2,  3,  and  4  of  this  report.   Chapter  2  is  a  guideline 
for  planning,  conducting,  analyzing,  and  reporting  on  an  HFE 
evaluation  according  to  the  requirements  of  DI-H-1334A.   Pro- 
cedures for  managing  the  HFE  evaluation,  allocating  test 
personnel^  developing  test  cost  estimates,  etc.,  are  also  con- 
tained in  Chapter  2.   This  information  will  aid  project  man- 
agers in  the  administration  and  organization  of  an  HFE 
evaluation.   The  explanations  of  the  DI-H-1334A' requirements 
will  assist  the  contract  monitor  to  understand  and  monitor  HFE 
tests  and  ensure  that  all  of  the  requirements  of  DI-H-1334A  are 
satisfied. 

Chapters  3  and  4  supplement  the  guidelines  given  in  Chapter 
2  by  giving  detailed  examples  of  HFE  reports,  written  according 
to  the  requirements  of  DI-H-1334A.   Chapter  3  presents  the 
HFE  test  report  of  a  system. in  the  experimental  prototype 
stage  of  development.   This  sample  HFE  report  focuses  on  de- 
terminining  the  feasibility  of  human  performance,  the  appro- 
priateness of  the  tasks  allocated  to  the  operator  and  to  the 
machine,  and  the  adequacy  of  the  workspace  layout.   This  report 
also  demonstrates  procedures  for  conducting  HFE  tests  and  mock- 
up  equipment. 

Chapter  4  contains  the  HFE  test  report  of  an  advanced 
development  prototype.   The  emphasis  in  this  stage  of  develop- 
ment is  on  determining  the  capability  of  the  operator  to 
perform  his  assigned  tasks  within  his  prescribed  time  and 
error  standards.   This  test  report  also  evaluates  the  adequacy 
of  operator  selection  and  training,  as  well  as  the  equipment 
configuration. 

Chapter  5  discusses  implications  of  human  performance  tests 
The  uses  to  which  data  can  be  applied  and  the  problem  associ- 
ated with  conducting  HFE  tests  are  also  described.   By  de- 
scribing how  HFE  test  data  can  be  applied  and  the  benefits 
of  collecting  the  data,  program  managers  can  better  appreciate 
the  need  for  HFE  testing." 
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RECOMMENDED  DOCUMENTS 


1.   MIL-STD-721B   Definitions  of  Effectiveness  Terms  for 
Reliability,  Maintainability,  Human  Factors  and  Safety 

"Applying  effectiveness  terminology  within  the  Department 
of  Defense  and  industry  requires  good  communications  and 
coordination  which,  in  turn,  requires  common  definitions  and 
terminology.   This  Standard  lists  and  defines  words  and  terms 
most  commonly  used  in  reliability,  maintainability,  human 
factors  and  safety.   Statistical  and  mathematical  terms  which 
have  gained  wide  acceptance  are  not  defined  in  this  Standard 
since  they  are  included  in  other  documents,  e.g.,  MIL-HDBK-217 . 

The  following  criteria  were  used  for  the  inclusion  of 
terms  and  definitions  deemed  pertinent  to  the  scope  of  this 
Standard: 

a.  Terms  and  their  definitions  which  are: 

(1)  Important  in  procurement  of  weapon  systems  for 
precise  definition  of  effectiveness  criteria. 

(2)  Unique  in  their  definition,  allowing  no  other 
meaning. 

(3)  Expressed  clearly,  preferably  without  mathematical 
symbols. 

(4.)  Fully  explanatory,  without  reference  to  any  other 
source  documents. 

b.  Terms  that  were  avoided  include: 

(1)  Those  found  in  ordinary  technical,  statistical, 
or  standard  dictionary  or  text  having  a  singularly  acceptable 
meaning  when  used  in  the  context. 

(2)  Terms  already  existing  in  other  Military  Standards 
outside  of  the  project  scope. 

(3)  Multiple  word  terms,  unless  needed  for  uniqueness. 

c.  The  purpose  of  this  Military  Standard  is  to  standardize 
on  meanings  of  terms  for  the  particular  application,  not  to 
compile  a  handbook. 
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2.   FED  STD  595  Colors 

"The  purpose  of  this  standard  is  to  present  in  covenient 
form  a  collection  of  standard  colors  currently  used  by  the 
various  departments  of  the  Government.   These  standard  colors, 
identified  by  5-digit  numbers,  are  defined  by  fundamental 
colorimetric  data.   For  reference  purposes,  each  color  is 
reproduced  within  a  1/2  x  1  inch  chip.   For  procurement,  in- 
spection and  other  color  matching  purposes,  3x5  inch  color 
chips  are  issued  as  specific  standards.   These  chips  are 
identified  by  the  same  5-digit  numbers. 

Spectrophotometric  curves  and  Commission  Internationale 
de  I'Eclairage  (CIE)  data  for  each  chip  are  included  as 
Volume  II  to  this  standard.   These  are  to  be  used  as  basic 
standards  for  reproducing  future  issues  of  the  chips,  also 
for  determining  changes  which  may,  or  may  not,  have  occurred 
in  chips  in  stock.   The  spectrophotometric  curves  and  CIE 
measurements  may  also  be  used  for  acceptance  testing  purposes 
in  lieu  of  the  3x5  inch  chips  if  so  specified.   However, 
accurate  comparison  can  be  made  only  if  values  and  curves 
are  obtained  on  the  same  instrument  standardized  under  the 
same  conditions." 


3.   Guide  to  Human  Factors  Engineering  General  Purpose  Test 
Planning  (GPTPj 

"This  report  is  concerned  with  human  factors  engineering 
test  and  evaluation  program  planning,  which  is  generalizable 
across  naval  weapons  systems  except  for  nuclear  weapons  and 
propulsion  subsystems.   The  Test  and  Evaluation  Program 
Planning  Guide  is  prepared  for  use  by  test  planners  and  is  in 
accordance  with  DOD  Directive  5000.3,  NAVMATINST  3900.0, 
NAVMATINST  3960.6,  OPNAVINST  3960.10,  and  OPNAVINST  4720.9. 
The  human  factors  engineering  Test  and  Evaluation  Program 
Plan  specifically  addresses  developmental  and  operational 
tests  as  well  as  production  acceptance  tests  to  provide  data 
for  satisfying  the  Defense  System  Acquisition  Review  Council 
milestones  for  system  acquisition." 


4.   HFTEMAN  Human  Factors  Test  ^  Evaluation  Manual,  Vols.  I, 
II,  and  ITI 

"In  most  military  systems  the  ultimate  effectiveness  of 
the  system  depends  on  the  capability  of  the  human  to  operate, 
maintain,  transport,  erect,  or  otherwise  use  the  system  equip 
ment. 
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In  the  test  and  evaluation  of  such  systems  adequate  con- 
sideration must  be  given  to  the  human  element. 

Human  Factors  Test  5  Evaluation  Manual  was  developed  to 
assist  Navy  and  Marine  Corps  Test  and  Evaluation  personnel  in 
the  evaluation  of  the  human  factors  aspects  of  equipment  items. 

Human  Factors  Test  5  Evaluation  Manual  is  primarily  di- 
rected toward  the  Human  Factors  Engineering  (HFE) .   The  detail 
provided'  for  the  Test  and  Evaluation  Plan: 

--  Guidelines  concerning  what  to  evaluate. 

--  Standards  or  criteria  to  evaluate  against. 

--  Planning  information  on  how  to  design,  setup,  conduct, 
and  analyze  data  from  a  human  factors  enginering  test 
or  evaluation. 

Human  Factors  Test  §  Evaluation  Manual  is  therefore  used 
when  the  detail  Test  and  Evaluation  plan  is  being  generated. 
It  enables  the  planner  to  identify  HFE: 

--  Test  objectives. 

--  Test  methods,  procedures  and  conditions. 

--  Test  measurements  to  be  acquired  and  recorded  during 
the  test  or  evaluation. 

--  Criteria  or  standards  against  which  measurements  are 
compared. 

--  Test  data  collection  materials  such  as  questionnaires 
or  checklists. 

--  Test  data  analysis,  presentation  and  reporting  require- 
ments . 

Human  Factors  Test  5  Evaluation  Manual  consists  of  three 
volumes : 

--  Volume  I  -  Data  Guide:  contains  the  guidelines  concern- 
ing what  to  evaluate  for  different  classes 
of  equipment  and  types  of  tests.  • 

--  Volume  II-  Support  Data:  contains  additional  criteria 
expanding  on  the  guidelines  contained  in 
Volume  I. 

--  Volume  Ill-Methods  and  Procedures:  contains  guidance  on 
how  to  design,  set  up,  conduct  and  analyze 
data  obtained  in  HFE  Test  5  Evaluation  program, 
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5.  Human  Factors  Engineering  Data  Guide  for  Evaluation  (HEDGE) 

"The  purpose  of  the  information  in  HEDGE  is  to  enable  you 
to  expand  your  test  capabilities  in  considering  the  human 
element.   They  will  provide  you  with  a  strategy  for  viewing 
any  item  which  is  undergoing  testing  from  the  standpoint  of 
the  soldier  who  must  ultimately  operate,  maintain,  or  other- 
wise utilize  it.   The  use  of  these  materials,  in  addition  to 
standard  Task  and  Design  Checklists  and  Questionnaires,  will 
enable  you  to  tailor  your  HFE  subtest  to  a  specific  item. 

These  materials  have  been  prepared  especially  for  you: 

a.  They  are  intended  to  support  test  engineers  not  design 
engineers . 

b.  They  were  designed  with  your  specific  tasks  in  mind, 
i.e.,  preparing  a  Test  Plan,  conducting  a  test,  analyzing 
and  interpreting  test  data,  and  generating  the  test  report. 

Because  these  materials  offer  you  a  strategy  for  conducting 
human  factors  testing  rather  than  a  compendium  of  facts,  the 
results  which  you  obtain  will  be  directly  proportional  to 
your  intelligent  and  common  sense  application  of  the  data 
presented." 

6.  Mission  Operability  Assessment  Technique:   A-  System 
Evaluation  Methodology  (MOAT) 

"The  MOAT  is  an  evaluation  methodology  that  measures  the 
operability  of  a  system  or  subsystem  in  terms  of  operator 
tasks  performed  during  a  mission.   MOAT  addresses  the  problem 
of  how  well  an  operator  can  use  a  system  or  subsystem  to  per- 
form tasks  within  the  mission  context.   Contrasted  to  evalu- 
ations using  human  engineering  design  criteria  which  present 
only  pass  or  fail  information,  this  technique  provides  infor- 
mation on  the  degree  of  system  and/or  subsystem  success  or 
failure.   This  report  discusses  how  MOAT  was  developed,  how 
MOAT  can  be  used  on  existing  or  emerging  systems,  and  how  MOAT 
will  be  expanded  in  future  development  to  include  multicrew 
station  evaluations.   This  report  examines  how  the  three  under- 
lying technologies  have  been  integrated  into  one  comprehen- 
sive methodology.   Task  analysis,  scaling  methodology,  and 
multi-attribute  utility  theory  are- discussed  in  terms  of  MOAT 
development  and  application.   Finally,  the  report  discusses 
the  importance  of  developing  systems  evaluation  methodologies 
which  provide  decision  makers  with  meaningful  information 
necessary  for  effective  decision  making." 
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APPENDIX  B. 

HUMAN  FACTORS  ENGINEERING  SELF- PACED  COURSE 
TASK  REQUIREMENTS 

Upon  completion  of  the  HFE  Course,  students  should  know: 

1.  The  meaning  of  common  terms  used  in  human  factors  engi- 
neering. 

2.  The  principal  HFE  references  (regulations,  standards, 
specification,  guides,  etc.)  and  where  in  them  to  look 
for  the  answers  to  specific  types  of  HFE  problems. 

3.  Potential  sources  of  technical  information  on  major  topics 
within  human  factors  engineering. 

4.  The  goals  of  HFE  in  a  materiel  development  program  (i.e., 
the  compatibility  of  the  four  factors:  human  performance 
requirements,  personnel  selection  criteria,  training  and 
equipment  design) . 

5.  When,  how,  and  for  what  purpose  in  the  general  scheme  by 
which  the  Department  of  Defense  (DOD)  develops  materiel 
human  factors  engineering  activities  should  be  performed. 

6.  How  to  determine  the  human  performance  requirements  in  a 
systems  concept. 

7.  • The  kinds  of  factors  and  forces  which  can  affect  human 

performance  and  how  to  identify  and  measure  them. 

8.  The  differences  between  measurements  taken  in  field  vs. 
laboratory  settings,  and  the  difficulties  of  generalizing 
laboratory  data  to  field  situations. 

9.  How  to  formulate  and  then  state  in  comprehensible  English 
performance  measures  applicable  to  a  specific  system  for 
the  general  dependent  variables  of  time  and  error. 

10.  What  "experimental  control"  measures  are  necessary  in  any 
test  involving  human  performance  and  the  probable  effects 
on  test  data  if  they  are  absent. 

11.  How  to  analyze  human  performance  data  (e.g.,  time  and 
error)  within  the  context  of  "system  effectiveness"  and 
"System  reliability". 
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12.  How  to  calculate  human  performance  reliability. 

13.  The  major  techniques  used  by  human  factors  engineers 
during  system  sythesis,  design  and  development. 

14.  How  to  recognize  a  good  task  analysis. 

15.  The  relationship  between  human  factors  engineering  and 
the  engineering  specialities  of  reliability,  maintain- 
ability and  safety. 

16.  How  to  interpret  and  apply  the  .standards  and  specifica- 
tions needed  for  human  factors  engineering  that  deal 
with  human  performance. 

17.  The  correct  method  of  stating  criteria  for  a  vehicle 
which  must  be  inaudible  100  meters  away  from  it,  and 
for  an  alarm  which  must  be  audible  for  500  meters. 

18.  How  to  determine  the  minimum  height  of  letters  on  a 
sign  which  must  be  visible  at  100  meters  on  a  sunny  day 

19.  How  to  interpret  an  analysis  of  variance  table  of  per- 
formance scores  in  an  experiment  having  threis  indepen- 
dent variables  with  one  statistically  significant  main 
effect  and  one  statistically  significant  first-order 
interaction. 
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APPENDIX  C. 

HUMAN  FACTORS  ENGINEERING  SELF- PACED  COURSE 
TERMINAL  LEARNING  OBJECTIVES 


Upon  completion  of  the  Human  Factors  Engineering  Self-Paced 
Course,  the  student  will  demonstrate  the  following  capabili- 
ties and  knowledge:. 

1.  An  understanding  of  common  terms  used  in  human  factors 
engineering. 

2.  A  familiarity  with  human  factors  references  and  an  ability 
to  use  them. 

3.  An  awareness  of  potential  sources  of  technical  information 
on  major  human  factors  topics. 

4.  An  understanding  of  the  goals  of  human  factros  engineering 
in  a  materiel  development  program. 

5.  An  ability  to  integrate  human  factors  priciples  in  a  DOD 
sponsored  program. 

6.  An  ability  to  determine  human  performance  requirements 
in  a  systems  concept. 

7.  An  understanding  of  the  kinds  of  factors  and  forces  which 
affect  human  performance  and  an  ability  to  identify  and 
measure  them. 

8.  An  awareness  of  the  differences  between  field  and  labora- 
tory measurements. 

9.  An  awareness  of  what  "experimental  control"  measures  are 
necessary  for  any  test  involving  human  performance  and 
the  effects  in  their  absence. 

10.  An  understanding  of  basic  statistical  techniques,  such 
as  an  analysis  of  variance. 

11.  An  ability  to  calculate  human  performance  reliability. 

12.  An  ability  to  formulate  performance  measures  for  the  de- 
pendent variables  of  time  and  error. 

13.  An  ability  to  analyze  human  performance  data  within  the 
context  of  "system  effectiveness"  and  "system  reliability", 
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14.  An  understanding  o£  the  major  techniques  used  by  human 
factors  specialists  during  system  synthesis,  design,  and 
development. 

15.  A  familiarity  with  task  analysis. 

16.  An  awareness  of  the  relationship  between  human  factors 
engineering  and  the  engineering  specialists  of  reliability, 
maintainability,  and  safety. 

17.  An  ability  to  interpret  and  apply  the  standards  and  speci- 
fications of  the  human  factors  engineering  community. 
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APPENDIX  D. 

HUMAN  FACTORS  ENGINEERING  SELF-PACED  COURSE 
LESSON  TOPICS  0UTLINES6 


Lesson  1;   Welcome  to  Human  Factors  Engineering 
I.   Course  Introduction 
II.   Importance  of  Human  Factors 

Lesson  2:   Why  Human  Factors? 

I.   Story  Line 

A.   Introduction  to  main  characters 

1.  Lt .  I .  M.  Eager 

2.  Capt.  B.  Smart 

II.   Human  Factors  Engineering 

A.  Definition  of 

B,  Historical  Perspective 

III.   Systems 

A.  Man-machine  Systems 

B.  Components  of  Man-machine  Systems 

Lesson  5:   Tragic  Mistakes  and  Positive  Consequences 

I.   Historical  Perspective 
A.   Examples 

II.   Common  Errors  -  Reading  and  Interpreting  Instruments 

III.   Common  Errors  -  Operating  Controls 

IV.   Current  Status  of  Human  Error 

V.   Man-Machine  Incompatibilities 

A.   Man-machine  Capability  Differences 


Extracted  directly  from  the  Student  Supplement  of  the 
Human  Factors  Engineering  Self -Paced  Course. 
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Lesson  4:   Basic  Information  Processing,  or  Is  Man  a  Machine? 

I.   Attention 
.  A.   Limits  to 

B.   Selective  Attention 

II.   Judgments 

A.  Absolute 

B.  Relative 

III.   Man-machine  Systems^ 

A.  Man  the  Processor 

B.  Stimulus-Response  Codes 

C.  Information  Transmission 

D.  Input-output  Processes 

Lesson  5:   History  and  Related  Technology,  or  Human  Factors, 
This  Is  Your  Life 

I.   History  -  HFE 

A.  Ancient  Cultures 

B.  Industrial  Revolution 

C.  World  War  II 

D.  Current  Efforts 

II.   Stage  of  HFE  Development 

A.  Pretechnology 

B.  Aerospace  '  • 

C.  Sociotechnical 

D.  Cosmopolitan 

III.   Review 


Lesson  6 :   Anthropometry,  or  Do  I  Fit? 

I.   Introduction- 

A.  Definition 

B.  Historical  Perspective 

II.   Design  Principles 

A.  Adjustability 

B.  5th  -  95th  Percentile  Range 
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Lesson  7:   Work  Space  Design  and  Arrangement,  or  Don't  Cramp 
My  Style 

I,   Work  Space  Envelope 

A.   General  Design  Principles 

II.   Analytical  Methods 

A.  Indexing/Rating  Method 

B.  Link  Analysis 

C.  Prototypes 

III.   Component  Location  and  Spacing  Problems 

Lesson  8:   Vision  Capabilities,  or  A  Shot  In  The  Dark! 
I.   Visual  Anatomy-Supplement 
II.   Visibility 

III.   Panel  Lighting 

A.  Flood  Lighting 

B.  Integral  Lighting 

IV.   Visual  Detection,  Identification,  and  Estimation 

Lesson  9:   Vision  Displays,  or  Are  My  Eyes  Deceiving  Me? 

I,   Types  of  Displays 

A.  Quantitative 

B.  Qualitative 

C.  Static 

D.  Dynamic 

II.   Scales 

A.  Digital 

B.  Fixed  Pointer 

C.  Moving  Pointer 

III.   Signal  and  Warning  Lights 
A.   Detection  Factors 

IV.   Alphanumeric  and  Symbolic  Characters 
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LESSON  10:   Auditory  Presentations,  or  When  Is  An  Alarm  Not 
An  Alarm? 

I,   General  Types  of  Displays 

A.  Displays  Using  Auditory  Modalities 

B.  Nature  of  the  Message 

II.   Sound  Characteristics 

A.  dB,  Hz. 

B.  Frequency,  Intensity 

C.  Human  Reception  Capabilities 

III.   General  Display  Principles 
IV.   Presentation  Principles 
V.   Warning  and  Alarm  Systems 

LESSON  11:   Standardization  of  Controls,  or  Which  Way  Is  Up? 

I.   Standard  Arrangement  -  Population  Stereotypes 

II.   Categorizing  Controls 

A.   Quantitative,  Qualitative,  Representational,  and 
Continuous 

III.   Functions  of  Controls 

A.  Continuous  Adjustment  Settings 

B.  Discrete  Settings 

IV.   Types  of  Controls 

A.  General  Types  -  Linear,  Rotary 

B.  Specific  -  Pushbutton,  Toggle  Switch,  Knobs,  etc 

V.   General  Rules  for  Selection 

A.   MIL-STD-1472C  and  MIL-HDBK-759 


LESSON  12 :   Positioning  of  Controls,  or  The  Right  Place  At  The  R 
Right  Time 

I.   Accidental  Activation  of  Controls 
A.   Prevention  Methods 
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II.   Location  of  Controls  and  Displays 

A.  Priority,  grouping  association  factors 

B.  Location-  and  body  position 

C.  Spacing  between  controls 


LESSON  15:   Control  Dynamics,  or  How  Hard  To  Twist  The  Knob 

I.   Design  Concepts 

A.   S-R  compatibility 

■B.   Control-display  ratio,  a  type  of  compatibility 

II.   Types  of  resistance 

A.  Static  friction 

B.  Sliding  friction 

C.  Elastic  resistance 

D.  Viscous  damping 

III.   Feedback 

A.  Intrinsic 

B.  Extrinsic 


LESSON  14:   Other  Senses,  or  Controls  That  Have  Shaped-Up 

I.   Review  lessons  11-13 

II.   Touch 

A.  Pressure  Sensitivity 

B.  Pain 

C.  Temperature 

III.   Touch  Coding 

A.  Shape  coding 

B.  Class  'A'  and  'B'  design 

C.  Texture  coding 

LESSON  15:   Vibration  and  Acceleration,  or  Take  It  'Easy  I  Have 
Weak  StomachI 

I.   Vibration 

A.  Definitions 

B.  Measurement 

C.  Body  Parts  affected 

II.   Whole  Body  Vibrations 

A.  Performance  effects 

B.  Tolerable  limits 
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III.   Acceleration 

A.   Performance  effects 
C.   Protective  measures 


LESSON  16:   Vigilance,  or  Stay  Awake  If  You  Can 

I.   Vigilance 

A.  Definition 

B.  Examples 

II.   Performance  Decrements  Due  to  Vigilance 

A.  Time  Frame 

B.  Magnitude  of  decrement 

C.  Display  characteristics 

III.   Signal  Characteristics 

A.  Noise 

B.  Signal  intensity 

C.  Rate  of  signal  presentation 

IV.   Other  Factors  Affecting  Watch-keeping  Behavior 

A.  Environmental  conditions,  noise 

B.  Atmospheric  temperature 

C.  Procedural  conditions,  work/rest  schedules 


LESSON  17:   Temperature  Effects,  or  Baby,  It's  Cold  Outside 

I.   Reactions  to  Temperature  Change 

A.  Physiological 

B.  Adaptation,  sensitization  and  habituation 

C.  Effective  temperature 

II.   Extreme  Cold 

A.   Physiological  and  performance  effects 


LESSON  18:    Atmospheric  Effects  or  I  Can't  Breathe 

I.   Description  of  Atmosphere 

A.   Gases,  density,  pressure 

II.   Hypoxia 

A.  Factors  causing 

B.  Effects  -  physiological  and  performance 

III.   CO2  and  CO 

A.  Physiological  and  performance  effects 

B.  Military  specif iciations 
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IV.   Radiation 

A.  Measurement  Definitions 

B.  Effects  of  overexposure 

V,   Prevention 

A.  Mask 

B.  Clothing 


LESSON  19:   Noise,  or  Can  You  Hear  Me? 

I.   Auditory  Anatomy 

A.  Structure 

B.  How  sound  travels  in  the  ear 

C.  Mechanical  and  electrical  transmission 

II.   Properties  of  Auditory  Stimuli 
A.   Frequency,  intensity 

III.   Communication 
A.   Maskers 

IV.   Physiological  Effects 

A.  TTS 

B.  PTS 

C.  Damage  risk  criteria 

V.   Protection  Against  Noise 
LESSON  20:   Review  of  Lessons  1-19 

LESSON  21:   System  Acquisition 

I.   Department  of  Defense  Acquisition  Policy 

A.  Variation  across  systems 

B.  0MB  A-109 

C.  DOD  5000.1 

II.   Phases  of  Weapons  Systems  Acquisition  Process 

A.  Concept  exploration 

B.  Demonstration  and  validation 

C.  Full-scale  development 

D.  Production  and  deployment 

III.   HFE  and  the  Acquisition  Process 
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LESSON  22:   System  Analysis,  or  The  Big  Picture 

I.   Purpose  of  System  Analysis 

A.  Scheduling 

B.  Identifying  limiting  factors 

C.  System  performance  criteria 

D.  Design  Options 

E.  Evalutation  of  Systems 

II.   Stages  of  System  Analysis 

A.  Requirements  analysis 

B.  Function  analysis 

C.  Task  analysis 

III.   Major  Problem  Areas 

A.  Subsystems'  interaction 

1.   Sectionalization  technique 

B.  Criterion  determination 

C.  Defining  human  performance  effectiveness 


LESSON  25:   Task  Analysis:  History  and  Perspectives 
I.   Recent  History  of  Task 

Definition  of  Task  Analysis 


II 
III 


IV. 


Other  Improtant  Task  Analysis  Factors 

A.  Systems  mission  and  function 

B.  Job,  task,  sub-task,  task  element 

C.  Task  inventory,  task  taxonomy 

Output  of  Task  Analysis 

A.  Design 

B.  Training 

C.  Test  and  evaluation 

D.  Manning 

E.  Workload 


LESSON  24:   Task  Analysis,  or  Fitting  Task  Analysis  into  the 
System 

I.   Stages  in  Task  Analysis  Process 

A.  Identify  task,  sub-task,  task  element 

B.  Develop  specific  behavioral  objectives 

C.  Identify  supporting  skills  and  knowledge 

II.   Task  Statements 

A.   Specific  behavioral  objectives 


128 


III.   Task  Analysis  Worksheet 

IV.   Sampling  Techniques 

A.  Activity  sampling 

B.  Process  Analysis 


LESSON  25:   Affordability ,  or  When  Can  We  Trade  Off  What? 

I.   Trade-off  analysis 

A.  Definition 

B.  Types: 

1.  Geometry  of  design 

2.  Manpower  allocation 

II.   General  System  Analysis 

A.  Operational  requirement 

B.  Hardware  design 

C.  Manpower  and  training  requirements 

D.  Saftey,  reliability , and  other  factors 

III.   Four  Major  Steps  in  Trade-off  Analysis 
A.   Baseline  Alternative 

1.  ROC 

2.  Initial  Hardware 

3.  Manpower  and  training  requirements 

IV.   Life-cycle  Costs 

A.  Computer  models 


LESSON  26:   Maintainability,  or  Can  Anybody  Fix  This? 

I,   Basic  Concepts 

A.  Maintainability 

B.  Maintenance 

C.  Reliability 

D.  Human  performance 

II.   Design  Features 

III.   Skill  Application 

A.  Capabilities/limitations 

B.  Training 

IV.   Predictions 
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LESSON  27 


Hazard  Analysis,  or  A  Stitch  in  Time 


I.   Introduction 

A.  HFE  and  safety 

B.  HPE  and   safety  design 

II.   Introduction  to  Hazard  Analysis 

III.   Identification  Phase 

A.  Checklists 

B.  Historical  records  -  intermediate  indicators 

C.  General  Investigations 

IV.   Evaluation  Phase 

A.  Grouping   according  to  category 

B.  Ranking  within  category 

V.   Cost  Countermeasures 


LESSON  28:   Training  the  Right  People 

I.   Selection  and  Training  in  Design/Conceptual  Phase 
A.   Approaches 

1.  Organizational  Analysis 

2.  Job  analysis 

3.  Person  analysis 

II.   Training  Techniques 

A.  On-the-job  training 

B.  Classroom  training 

C.  Computer-aided  instruction 


D. 

Team  Training 

LESSON 

29: 

Does  Training 

I. 

ISD 

Phases 

A. 

Analysis 

B. 

Design 

C. 

Develop 

D. 

Implement 

E. 

Control 

II.   Validity  and  Reliability 
A.   Types  of  validity 

1.  Training 

2.  Performance 

3.  Intra-or-zanizational 


4. 


Inter- organizational 
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III.   Simulator  Training  Issues 

LESSON  30:   Psychophysical  Methods,  or  Do  I  Detect  a  Signal? 

I.   Psychophysics 

A.  Definition 

B.  Purpose 

II.   Classical  Methods 

A.  Methods  of  limits,  adjustment  and  constant  stimuli 

B.  Assumptions 

C.  Advantages,  disadvantages 

III.   Theory  of  Signal  Detection 

A.  Response  bias 

B.  Decision  matrix 

IV.   Scaling  Methods 

A.  Direct 

B.  Indirect 


LESSON  51:   Experimental  Methods,  or  How  Do  I  Know  If  I've 
Done  It  Right 

I.   Eager *s  Experiment 

II.   Research  Categorization 

A.  Theoretical 

B.  Empirical 

1.  Observation,  correlation,  experimenttation 

2.  Natural  (field),  laboratory 

III.   Variable  Classifications 

A.  Qualitative  and  quantative 

B.  Independent  and  dependent 

C.  Relevant 

1.  Subject 

2.  Situational 

3.  Sequence 


LESSON  52:   Experimental  Methods,  or  How  Do   I  Control  These 
Influences? 

I.   Control  Techniques 

A.  Subject 

B.  Situational 

C.  Sequence 
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II.   Designs 

A.  Within  subject 

B.  Between  subjects 

1.  Simple  randomized 

2.  Factorial 

III.   Validity 

A.  Internal 

B.  External 


Lesson  35:   Statistics,  Part  I 

I.   Introduction 

A.  Description  of  total  statistics  lessons 

B.  Description  of  statistics 

C.  Purpose  of  statistics 

II.   Frequency  Distributions 

A.  Ungrouped  data 

B.  Internal  data 

C.  Meaning  of  any  one  score 

D.  Percentile  ranks 

III.   Measures  of  Central  Tendency 

A.  Mean 

B.  Normal  curve 

C.  Median 

IV.   Measures  of  Despersion 

A.  Mean  deviation 

B.  Standard  deviation 

C.  Variance 

D.  Relation  of  50  and  normal  curve 


LESSON  54:   Statistics,  Part  II 

i.   Correlation 

A.  Degree 

B.  Direction 

II.   Inferential  Statistics 

A.  Nonparametric 

B.  Parametric 

C.  Appropriate  statistical  tests 

1.  t-test 

2.  F  test 
5 .   ANOVA 
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III.  Interpretation  of  Results 

A.  Practical  vs.  statistical  significance 

B.  One-way  AN OVA 

C.  Two-way  ANOVA 

D.  Higher  order  ANOVAs 

LESSON  35:   Review,  or  How  Have  I  Done  So  Far? 

LESSON  36:   Human  Factors  and  the  Military 
I.    AR  602-1 
II.    MIL-STD-1472C 
III.    MIL-HDBK-759 

IV.  MIL-STD-1474B 


LESSON  37:   Human  Factors  Test  and  Evaluation,  or  Can  a 
HFTEMAN  Cross  a  MOAT  using  a  HEDGE? 

I.    HF  Testing 

A.  HEL  TM  29-76 

B .  MOAT 

C .  HFTEMAN 

D.  HEDGE 

E.  HRTES 

II.    Human  Performance  Measures 

A,   Analysis  of  human  performance  data 


LESSON  38:   Real  World  Problem,  Part  I 


LESSON  39:   Real  World  Problem,  Part  II 


LESSON  40:   Overall  Summary 
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APPENDIX  E 

EVALUATION  PLAN  FOR  THE 
HUMAN  FACTORS  ENGINEERING  SELF-PACED  COURSE 
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Acronyms  and  Abbreviations 

COTF     Commander  Operational  Test  and  Evaluation  Force 

CCS      Command  Course  Supervisor 

DCOTF    Deputy  Commauider  Operational  Test  and   Evaluation 
Force 

HP  Human  Factors 

HFE  Human  Factors  Engineering 
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NFS  Naval  Postgraduate  School 
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SIF  Student  Information  Form 
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7 
Several  of  the  references  used  to  design  the  test  plan 

have  been  revised  and  incorporated  into  the  main  body  of  the 

thesis.   The  criteria  and  standards  sure  in  accordance  with 

the  updated  versions  of  the  above  instructions. 
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Section  I 
Introduction  to  the  Evaluation  Plain 

101.  Pur-pose.   The  purpose  of  this  evaluation  plan  is  to 
assess  the  operational  effectiveness  of  the  Human  Factors 
Engineering  (HFE)  Self -Paced  Course  and  its  potential  value 
for  use  in  the  military  test  and  evaluation  community. 

102.  Course  Description.   This  course  is  divided  into  three 
major  sections,  Section  I  (Lessons  1-5)  deals  with  the 
human's  capabilities  and  limitations.   In  this  section,  the  1. 
history  and  continued  need  for  human  factors  engineering  will 
be  investigated.   Lessons  6-10  are  concerned  with  the  physio- 
logical capabilities  and  limitations  of  human  beings.   Lessons 
11-13  discuss  how  the  proper  design  of  controls  and  displays 
makes  use  of  the  information  learned  in  the  previous  lessons. 
Lessons  1^-19  talce  a  loolc  at  human  interactions  with  the 
environment,  followed  by  a  review  of  the  first  section  in 
lesson  20. 

The  primary  focus  of  Section  II  is  upon  the  human  fit- 
ting into  the  system.   Lessons  21-27,  investigate  the  role  of 
the  human  engineer  in  various  aspects  of  analysis,  such  as 
systems  amalysis,  cost  analysis,  task  analysis,  etc.   Lessons 
28-29  are  concerned  with  the  selection  and  training  of  person- 
nel.  In  lessons  30-3^ »  the  student  will  receive  a  short  course 
on  experimental  techniques  and  statistical  concepts.   Finally, 
lesson  35.  will  offer  a  review  of  Section  II. 
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Section  III  is  entitled  "Human  Factors  in  the  Military" . 
In  this  section,  lessons  36   and  37  focus  on  human  factors 
organizations,  documentation,  and  future  application. 

In  order  to  give  the  students  a  practical  application  of 
what  has  been  presented  in  the  course,  lessons  38  and  39  will 
ask  him/her  to  work  on  a  "real  world"  problem.  Finally,  in 
lesson  ^0,  a  more  typical  review  of  the  entire  course  is 
presented. 

103.   Baclcground.   The  HFE  Self -Paced  Course  was  developed  to 
satisfy  a  need  for  increased  awareness  auid  more  indepth  under- 
stamding  of  Human  Factors. 

Reference  (a),  highlights  this  need  amd  identifies  a 
deficiency  in  the  performance  of  various  weapons  systems 
"...because  the  DOD  does  not  pay  enough  attention  to  logistic 
support,  human  factors  and  quality  assurance  during  the  design 
phase  of  the  acquisition  process.  These  problems  deter  the 
systems*  effectiveness  to  defend  our  country  in  case  of  war." 
The  Government  Account  Office  (GAO)  "therefore  makes  recommen- 
dations to  improve  the  management  and  planning  of  ownership 
considerations  that  have  an  impact  on  the  effectiveness  of  a 
weapon  system.** 

The  Human  Factors  Test  and  Evaluation  Manual  (HFTSMAN), 
Vols.  I,  II  and  III  (reference  (d))  was  distributed  to  various 
government  agencies  in  October  1976  by  Pacific  Missile  Test 
Center,  Pt.  Mugu,  California.   HFTEMAN  was  developed  to  provide 
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standardization  in  procedures,  testing  and  criteria  in  eval- 
uating human  factors.   It,  however,  assumes  a  basic  knowledge 
of  hiunan  factors  for  its  most  effective  use. 

The  HFE  Self -Instruct ion  course  was  developed  in  order  to 
provide  this  basic  Imowledge.   It  has  not  been  evaluated  or 
used  on  a  trial  basis  in  any  portion  of  the  military  prior  to 
this  time. 
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Section  II 
Administrative  Information 

201.  General.  General  responsibilities  for  activities  in- 
volved in  this  evaluation  are  provided  in  this  section,  as  well 
as  appropriate  points  of  contact.  Continuing  close  liaison  is 
essential  to  timely  and  successful  prosecution  of  this  evalu- 
ation. 

202.  Responsibilities . 

a.  Naval  Postgraduate  School  (Lt.  M.  M.  Fleming) 

(1)  Promulgate  major  changes  to  this  evaluation  plan. 

(2)  Coordinate  arrangements  for  HPE  course  subjects. 

(3)  Coordinate  distribution  of  required  course 
materials  to  participants. 

(k)     Conduct  briefings  for  all  participating  if  so 
requested  and  funded  by  the  requesting  activity. 

(5)  Provide  certificates  of  completion  or  equivalent. 

(6)  Analyze  evaluation  results  and  make  them  avail- 
able to  the  appropriate  units  upon  request. 

b.  COMCPTEVFOR,  DEPCOMOPTEVPOR,  APT&E  Center/TELH,  TESTG/ 
ENAH 

(1)  Furnish  names  and/or  Student  Information  Form 
(SIF)  numbers  of  participating  students, 

(2)  Provide  point  of  contact  for  test  subjects  with- 
in each  command  (Command  Course  Supervisor  -  CCS). 

(3)  Keep  Lt.  M,  Fleming  advised  of  students  progress. 
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c.   U.  S.  Army  Human  Engineering  Laboratory,  Pacific 
Missile  Test  Center  (PMTG)i 

(1)  Provide  required  number  of  copies  of  all  course 
materials  for  distribution  to  subject  students. 

(2)  Provide  point  of  contact  to  Lt.  Fleming. 

(3)  Provide  required  plans,  schedules  and  procedural 
guidelines  as  indicated  in  Section  IV. 

203.   Points  of  Contact. 

a.  Naval  Postgraduate  School 

Lt.   Martha  M.    Fleming 
Evaluation  Director 
Naval  Postgraduate  School 
SMC  #  13^0 

Monterey,   California     93940 
Autovon;      878-2536 
Telephone!     ^4-08-646-2536 

Cdr.    Bill  Moroney/  Dr.    R.   A.    McGonigal 

Thesis  Advisor 

Naval  Postgraduate  School 

Code  b^   MP 

Monterey,  California  939^0 

Autovon:   878-2620/2594 

Telephone:   408-646-2620/2594 

Professor  Richard  S.  Elster 
Secondary  Thesis  Advisor 
Naval  Postgraduate  School 
Code  5^Ea 

Monterey,  California  93940 
Autovoni   878-2792 
Telephone:  408-646-2792 

b.  COMOPTEVFOR 

Lt.  Vickie  Bonnano 
Command  Course  Supervisor 
COMOPTEVFOR,  Code  02A 
Naval  Station 
Norfolk.  Virginia  23511 
Autovon:  690-5598 
Telephone:   804-444-5598 
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c.  DEPCOMOPTEVFORPAC 

Capt.  Robert  E.  Sheridan 

Command  Course  Supervisor 

DEPCOMOPTEVFORPAC 

NAS  North  Island 

San  Diego,  California  92135 

Autovon;   951-6970 

Commercial: 

d.  AFT&E  Center/TELH 

LCOL  P.  A.  Crowley 

Command  Course  Supervisor 

AFT&E  Center/TELH 

Kirkland  AFB,  New  Mexico  87117 

Autovon:   244-9606 

Commercial: 

e.  TESTG/ENAH 

Cyrus  T.  Crites 

Command  Course  Supervisor 

2620  TESTG/ENAH   Stop  239 

Edwards  AFB,  California 

Autovon:   350-3334 

Commercial: 

f.  Pacific  Missile  Test  Center,  U.  S.  Army  Human 
Engineering  Laboratory 

Cdr.  Tom  Jones 

Pacific  Missile  Test  Center 

Code  1226 

Pt.  Mugu,  California   93042 

Autovon:   351-8981 

Commercial:   805-982-8981 

g.  ASD/ENECH 

Dr.  Richard  Shiffler 

asd/enech 

Wright  Patterson  AFB,  Ohio  43433 

Autovon:   785-6010 

Commercial: 


h.   Capt.  Don  Loose 

Electronics  Systems  Division 
Hanscom  AFB,  Massachusetts  OI73I 
Autovon:   478-2825 
Commercial: 
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Section  III 
Scope  of  the  Evaluation 

301.   OTD.iectives.   In  accordance  with  the  recommendations  of 
reference  (a),  standardized  objectives  upon  which  this  course 
is  to  be  evaluated  will  follow  the  procedures  found  in  refer- 
ences (c)  through  (j)  and  as  follows: 

a.  Course  Objectives.   The  course  objectives  must  be 
developed  according  to  accepted  stajidardized  structure  in  order 
to  prepare  the  student  to  perform  his/her  job  in  testing  and 
evaluating  human  factors. 

b.  Course  Material  Design.   The  format  of  the  course 
materials  must  aide  the  student  in  accomplishment  of  the  stated 
learning  objectives.   The  following  areas  are  included  in  this 
evaluation I 

1.  Sequence  and  structure 

2.  Figures  and  Tables 

3.  Symbol ogy  and   legends 
k.      Layout  and  format 

c.  Course  Content.  The  course  content  must  support  the 
course  learning  objectives. 

d.  Course  Presentation.   The  design  of  the  course  pre- 
sentation aids   the  student  in  accomplishment  of  the  course 
learning  objectives  without  the  aid  of  an  on-site  instructor. 

In  addition  to  the  above,  future  plans  for  the  following 
will  be  reported! 

a.   Course  Availablilitv.   The  course  materials  should  be 
maintained  in  sufficient  quantity  to  meet  the  needs  of  those  it 

will  serve. 
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b.  Course  Maintainability.   The  course  materials  must  be 
in  accordance  with  the  most  recent  version  of  the  references 
upon  which  it  is  based  and  humsm  factors  design  developments. 

c.  Course  Supportability.  Assistance  must  be  available 
by  correspondence  with  the  issuing  agency.   Feedback  must  be 
provided  to  the  student  on  his/her  performance.   The  life  cycle 
cost  of  the  course  must  be  available  for  future  planning  and 
assessment. 

302.   Criteria.  The  criteria  below  are  in  accordance  with  ref- 
erences (e)  through  (i)  and  other  resource  materials  listed  in 
the  bibliography. 

a.   Course  Ob.iectives.   The  course  learning  objectives 
should  have  all  the  characteristics  listed  below. 

1.  Objectives  must  be  a  statement  of  student  behavior 
(action),  such  as  the  creation  of  a  product  or  some  other  overt 
act,  which  can  be  accepted  as  evidence  that  the  intended  out- 
come has  occurred. 

2.  The  behavior  must  describe  specifically  all  out- 
comes that  will  demonstrate  that  learning  has  occurred. 

3.  The  student  behavior  called  for  must  be  capable 
of  observation  sind  evaluation  within  the  learning  and  testing 
environments. 

k.   The  objective  must  be  stated  in  learner  rather  than 
teacher  terms,  i.e.,  actions  which  the  student  will  perform 
rather  than  what  the  instructional  materials  will  "say  or  do". 

5.  There  must  be  a  standard  against  which  the  student 
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behavior  will  be  measured.   It  must  be  fully  specified. 

6.  The  statement  of  the  conditions  under  which  the 
student  behavior  will  occur  must  be  fully  specified. 

Additionally,  students  must  be  able  to  satisfy  the  ob- 
jectives during  the  normal  performance  of  their  jobs  in  human 
factors  test  and  evaluation. 

b.  Course  Material  Design. 

1.  Sequence  and  structure.  The  learning  objectives 
should  be  arranged  in  the  sequence  in  which  instruction  will  be 
presented  to  the  student. 

2.  Figures  and  Tables.  Sufficient  information  should  be 
provided  with  each  figure  aind  table  to  allow  the  student  to 
apply  it  as  directed  within  the  sourse  materials  smd  during 
performance  of  his/her  job  when  left  only  with  the  student 
supplement. 

3.  Symbol ogy  and  Legends.  The  symbols  and  legends 
presented  in  the  course  materials  should  be  implicit 

and  standardized  where  possible. 

k.    Layout  amd  format.  The  layout  and  format  must  aide 
the  student  in  accomplishment  of  course  objectives  and  encourage 
motivation  and  desire  to  complete  the  course. 

c.  Course  Content.  The  mean  score  of  the  course  groups* 
second  diagnostic  tests  must  be  at  least  one  standard  deviation 
above  the  meam  score  of  the  control  groups*  second  diagnostic  test. 

d.  Course  Presentation.  The  instructions  and  verbal  context 

of  the  course  materials  must  allow  accomplishment  of  course 

objectives  and  course  completion  without  the  aide  of  an   on-site 

instructor. 
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303.   Evaluation. 

a.  Site  selection.   The  evaluation  of  the  HFE  course  will 
be  conducted  at  various  evaluation  sites  which  are  involved  in 
test  and  evaluation  of  systems  involved  in  the  acquisition 
cycle.   Testing  at  these  sites  will  provide  a  realistic  environ- 
ment in  which  to  exercise  the  course.   The  course  is  designed 
for  personnel  with  this  same  job  type  and  responsibilities. 

b.  Personnel  selection.  Subjects  participating  in  this 
evaluation  will  be  drawn  from  three  military  commands  whose 
primary  mission  is  operational  test  and  evaluation.   Some  per- 
sonnel will  be  volunteers,  while  others  may  be  assigned  to 
participate. 

1.  Specialty  area.   It  is  smticipated  that  the 
evaluation  subjects  will  be  drawn  from  the  various  divisions 
within  each  command  (air,  surface,  subsurface  and  special  war- 
fare). 

2.  Special  requirements.  No  constraints  regarding 
rate,  rank,  AFS,  grade,  educational  level  or  prior  experience 
will  be  placed  on  participants.   The  only  requirement  is  that 
each  subject  be  actively  involved  in  plamning  £Lnd  performing 
operational  test  and  evaluation. 

c.  Data  Collection.  Data  sheets,  course  diagnostic 
tests  and  questionnaires  for  use  in  this  evaluation  are  con- 
tained in  Annex  A.   Copies  will  be  distributed  to  Command 
Course  Supervisors  prior  to  beginning  the  evaluation.  IViore 
specifics  regarding  the  actual  evaluation  procedures  are  pro- 
vided in  Section  IV. 
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304.   Limitations  to  Scope.  While  the  HFE  Self-Instruction  Course 
consists  of  40  lessons,  only  the  first  20  are  scheduled  to  be 
evaluated.  All  activities  taking  part  will  be  provided  with 
all  40  lessons  in  order  to  provide  subjects  the  opportunity  to 
complete  the  course  at  a  pace  faster  than  that  recommended  by 
Pacific  Missile  Test  Center. 

The  criteria  established  for  assessing  the  adequacy  of  the  course 
objectives  require  a  prolonged  period  for  a  full  and  complete 
evaluation.   This  evaluation  lasts  only  for  a  period  of  30 
days  2uid  will  only  analyze  the  completed  questionnaires  of  the 
participating  subjects.   This  will  only  provide  a  preliminary  and 
limited  assessment.   Therefore,  it  is  recommended  that  a  second 
questionnaire  be  drafted  amd  distributed  to  the  course  par- 
ticipants six  months  after  course  completion.   The  subjects  then 
will  have  had  am  opportunity  to  apply  what  the  course  taught 
them  and  make  a  more  accurate  assessment  of  of  its  value. 
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Section  IV 
Evaluation  Guideline 

401.  Evaluation  participants  sind  H?£  Course  Materials.   The 
HFE  course  is  designed  to  provide  Human  Factors  training  to 
personnel  responsible  for  planning  and  conducting  testing  in 
human  factors.   This  is  part  of  the  job  of  sun  Operational  Test 
Director  (OTD)  as  stated  in  references  (c)  and  (j).  The 
personnel  aiding  in  the  evaluation  of  this  course  should  therefore 
have  the  same  responsibilities.   This  is  the  reason  for  the 
selection  of  the  particular  evaluation  sites.   This  course 

was  designed  for  the  job  and  is  not  limited  as  to  rate,  rank. 
Air  Force  Specialty  (AFS),  grade,  educational  level  or  prior 
experience  of  the  participaint. 

402.  Evaluation  Procedures.   The  HFE  course  will  be  conducted 
as  follows: 

a.  Student  Information  Forms  (SIF's)  must  be  filled  out 
by  the  CCS  for  each  course  participant  (See  Annex  A).  At  that 
time,  each  student  will  be  assigned  a  4-digit  code.  The  first 
two  digits  (from  the  left)  identify  each  individual  command; 

COMOPTEVFOR     OlXX 

DEPCOMOPTEVFOR  02XX 

AFT&E  Center    03XX 

TESTG/ENAH      04XX 
The  last  two  digits  are  then  assigned  in  numerical  sequence  to 
each  participating  student: 

Student  1       XXOl 

Student  2       XX02 
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Example*   COMOPTEVFOR  —  3  students 

Student  1        0101 

Student  2        0102 

Student  3       OIO3 
This  HPE  SIP  course  number  should  be  retained  by  both  student 
and  supervisor.   It  should  appear  on  all  correspondence,  tests, 
questionnaires  amd  SIF's  of  each  student. 

b.  After  an   SIF  has  been  completed  and  prior  to  beginning 
the  course,  each  student  will  be  given  a  diagnostic  test,  to 
determine  the  individual  level  of  each  student  before  beginning 
the  course.   Each  CCS  will  administer  the  test  amd  make  the  pur- 
pose of  the  test  clear  to  each  participant. 

In  addition,  the  student  must  be  instructed  not  to  guess  if  an 
amswer  to  a  particular  question  is  not  known  —  the  answer 
should  be  left  blank.   The  tests  will  be  mailed  back  to  NFS 
immediately  upon  completion  for  retention  and  evaluation. 

c.  Following  the  diagnostic  test,  each  student  will  be 
provided  with  the  course  materials  aind  references.   A  schedule 
for  timely  completion  of  the  course  will  also  be  provided.   Upon 
completion  of  the  course,  a  final  diagnostic  test  will  be  ad- 
ministered to  each  student,  by  the  CCS.   This  test  is  also 
designed  around  the  course  objectives.   While  the  questions  are 
not  the  same  as  the  first  diagnostic  test,  they  will  basically 
cover  the  same  material.   This  test  is  to  be  mailed  back  to  NFS  for 
scoring.   The  performance  of  the  student  on  each  of  the  diagnos- 
tic tests  will  then  be  compared.   Should  the  participant  wish  to 
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to  know  his/her  score  on  the  test(s),  it/they  will  be  provided 
by  the  Evaluation  Director  at  NFS  upon  request. 

d.  A  questionnaire  will  be  provided  to  each  student  who 
begins  the  course  to  provide  feedback  on  individual  opinions 
about  the  course  after  each  has  completed  it. 

e.  A  questionnaire  will  also  be  provided  to  each  CCS. 

f.  Questions  asked  by  the  student  should  be  recorded 
by  the  CCS  in  the  provided  log.   Each  question  should  also  be 
identified  with  an  SIF  number. 
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^03.   Test  01.   Course  Ob.iectives. 

a.  Ob.iect.   To  assess  the  extent  to  which  the  course 
objectives  support  the  tastes  required  of  a  human  factors  test 
and  evaluation  agent, 

b.  Procedure.   A  taste  analysis  of  the  human  factors 
specialist  will  be  compared  to  the  course  objectives.   Ques- 
tionnaires regarding  the  course  objectives  will  be  distribu- 
ted to  vairious  resources  dealing  with  human  factors  test  and 
evaluation  and  curriculum  design, 

c.  Data  Requirements.   A  taste  analysis  of  a  human 
factors  test  and  evaluation  specialist  and  completed  question- 
naires are  required. 

d.  Data  Analysis.   Course  objectives  will  be  compared 
to  the  taste  analysis  to  evaluate  how  adequately  the  course 
objectives  support  the  person  responsible  for  performing 
human  factors  test  and  evaluation. 

^0^.   Test  02.   Course  Materials  Format. 

a.  Object.   To  assess  readibility,  standardization  and 
clarity  of  format  according  to  references  (e)  through  (i). 

b.  Procedure.   Questions  directed  to  the  CCS  by  each 
participant  relative  to  the  HFE  course  for  clarification  or 
explanation  will  be  recorded.   Questionnaires  will  be  provi- 
ded to  students  and  CGSs  upon  course  completion  and/or  end  of 
test  period,  which  ever  applies, 

c.  Data  Requirements.   All  recorded  student  questions, 
questionnaires  and  comment  sheets  will  be  forwarded  to  the 
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to  the  Evaluation  Director  at  NPS,  upon  completion  or  end  of 
test  period  which  ever  is  applicable. 

d.   Data  Analysis.   Questionnaire  responses  will  be 
evaluated  suid  the  results  tabulated.   (See  Annex  B  for  further 
details. ) 

^05.   Test  03.   Course  Content. 

a.  Object.   To  assess  the  extent  to  which  the  course 
content  supports  the  course  objectives, 

b.  Procedure.   Two  diagnostic  tests  will  be  administer- 
ed to  each  student;  one  prior  to  beginning  the  course  and  one 
following  its  completion.   A  questionnaire  will  also  be  provi- 
ded at  course  completion.   The  SIF  number  will  be  placed  on 
each. 

c.  Data  Requirements.  Participant  test  answer  sheets 
and  completed  questionnaires  must  be  returned  to  the  evalua- 
tion director  at  NPS  as  they  are  turned  in  to  each  supervisor. 

d.  Data  Analysis.   A  mean  and  its  standard  deviation 
will  be  computed  for  each  of  the  diagnostic  test  scores.   The 
mean  scores  will  then  be  compared.   The  individual  responses 
to  test  questions  will  also  be  compared. 

Both  diagnostic  tests  will  also  be  administered  to 
a  control  group.   These  individuals  will  not  be  talcing  the 
course.   The  purpose  will  be  to  substantiate  the  reliability 
of  the  tests,  and  to  assess  whether  the  scores  of  the  course 
participants  change  more  than  do  scores  of  the  control  group. 
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Individual  questionnaires  will  also  be  evaluated  and 
analyzed,   (See  Annex  B  for  further  details). 

^06.   Test  0^.   Course  Presentation. 

a.  Ob.iect.  To  assess  the  clarity  of  the  presentation 
of  course  content,  in  order  to  allow  the  student  to  complete 
the  course  without  the  aid  of  an  on-site  instructor. 

b.  Procedure.   See  Test  02. 

c.  Data  Requirements.   See  Test  02, 

d.  Data  Analysis.   See  Test  02. 

^07.   Test  05.   Course  Availability. 

a.  Ob .1  ec t .   To  report  on  the  plans  for  making  course 
materials  available  to  future  students.   (Implementation  and 
Control ) . 

b.  Procedure.   Review  the  plans  PMTC  would  recommend 
for  making  course  materials  available  for  general  use. 

c.  Data  Requirements.   Provision  of  the  implementation 
and  controls  plans  from  the  issuing  agency  will  be  required. 

d.  Data  Analysis.  The  response  of  the  issuing  agency 
will  be  reported.   No  other  analysis  is  intended, 

408.   Test  06.   Course  Maintainability. 

a.  Ob.iect.   To  report  on  the  plans  for  keeping  the 
instruction  manuals  up  to  date  with  the  applicable  military 
standards  it  references  and  latest  human  factors  developments 
and  procedures. 

b.  Procedure.   Review  the  plans  of  the  issuing  agency 
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for  scheduling  updates  in  order  to  keep  the  course  content 
current . 

c.  Data  Requirements.   Provision  of  the  course  main- 
tenance plans  will  be  required  of  the  issuing  agency. 

d.  Data  Analysis.   The  response  of  the  issuing  agency 
will  be  reported.   No  other  analysis  is  intended. 

409.   Test  07.   Course  Supportabilitv. 

a.  Ob.iect.   To  assess  the  plans  and  procedures  for 
providing  adequate  support  to  students  taking  the  HPE  Self- 
Paced  Course. 

b.  Procedure.   Review  the  plans  and  procedures  provid- 
ed by  the  issuing  agency  for  supplying  the  support  required 
by  the  student  for  successful  completion  of  the  course. 

c.  Data  Requirements.  The  issuing  agency  will  pro- 
vide the  required  documents  to  the  NPS  Evaluation  Director 
for  review. 

d.  Data  Analysis.   See  Test  06. 
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Section  5 
Report(3) 

501.   General .   The  results  of  this  HPE  Self -Paced  Course 
Evaluation  will  be  provided  in  the  Evaluation  Director's 
thesis. 
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ATJNEX  3 


CCS   LOG   FORI..AT 
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INTRODUCTION 
FOR 
COWHAND  COURSE  SUPERVISOR  (CCS) 

You  have  been  identified  by  your  command  to  be  my  point 
of  contact  with  your  command  in  the  evaluation  of  the  Human 
Factors  Engineering  Self-Instruction  Course  during  the  period 
1  September  -  31  October  1981.   The  actual  length  of  the 
evaluation  at  your  command  may  vary  depending  on  how  quickly 
the  individual  student  participants  complete  the  course. 

In  reading  the  CCS  Guidelines,  you  will  find  that  your 
responsibilities,  very  basically,  consist  ofi 

(1)  Filling  our  Student  Information  Forms  (SIF) 
a.   Assigning  SIF  identification  numbers. 

(2)  Distributing  and  collecting  diagnostic  tests/ 
questionnaires,  comments  amd  course  materials. 

(3)  Maintaining  your  CCS  log 

a.  Student  Progress  Record 

b.  Student  Question  and  Comment  Record 

c.  Student  Questionnaires 

d.  CCS  Questionnaires 

e.  Evaluation  Completion  Checklist 

(4)  Contacting  me  for  the  amswers  to  any  questions 
which  the  student  participant  may  ask  that  are  not  obvious 
to  you  or  have  not  been  provided  in  the  material  sent 
you.   My  phone  number  isj   Autovoni   878-2536 
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(5)  Returning  all  materials  and  correspondence  as 

each  is  completed i  but  no  later  than  1  November  1Q81. 

Mail  to  I 

Lt.  Martha  M.  Fleming 
Naval  Postgraduate  School 
SMC  #  1340 
Monterey,  Cal.   939^0 
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EVALUATION  GUIDELINES 

FOR  THE 

COMMAND  COURSE  SUPERVISOR  (CCS) 

1.   Student  Information  Forms  (SIF)  and  Diagnostic  Test  #1; 

It  is  recommended  that  you  arrange  to  meet  with  participants  in 

a  group. 

a.  Before  the  meeting,  it  is  essential  that  youi 

(1)  Read  the  SIF  Instruction  sheet  to  be  sure  that  you 
linderstand  what  is  wanted  in  each  question. 

(2)  Fill  in  the  SIF  numbers  on  each  form. 

b.  During  the  meeting i 

(1)  Explain  what  the  evaluation  is  all  about.  Hand  out  SO-l/4. 

(2)  Hand  out  the  "Privacy  Act  Statement"  (DA  Form  ^368-R, 
1  May  1975) »  which  is  attached  to  the  front  of  the  SIF.   This 
form  basically  promises  them  confidentiality  of  the  information 
they  provide  in  the  SIF  and  the  questionnaires.   As  long  as  they 
follow  directions  provided,  only  the  participant  and  the  Eval- 
uation Director  at  the  Naval  Postgraduate  School  will  see  their 
responses  to  questionnaires  and  test  results.   As  it  states  at 
the  bottom  of  the  form,  they  may  keep  DA  Form  ^368-R,  but  do 
need  to  return  the  attached  SIF. 

(3)  SIF J   Point  out  that  "name"  is  optional.   The  rest 
should  be  self-explanatory.   If  not,  you  have  the  SIF  Instruction 
Sheet  which  should  provide  you  with  amy  necessary  clarifying 
information. 
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ik)     Diagnostic  Test  Number  1»   After  the  SIF  has 
been  collected! 

(a)  Pass  out  the  "Diagnostic  Test  Number  1". 

(b)  Read  through  the  instructions  with  the 
students  orally.   Be  sure  to  highlight  those  statements  or 
words  which  are  underlined. 

(c)  Offer  them  the  opportunity  to  take  the 
test  in  a  room  where  it  is  nice  and  quiet;  where  they  won't 
be  disturbed.   They  may  take  the  test  back  to  their  desks  or 
spaces,  which  ever  they  would  feel  most  comfortable  doing. 
Even  though  the  test  is  designed  to  see  how  much  they  know 
about  the  world  of  human  factors  before  taking  the  course, 
the  test  is.  not  going  to  be  graded,  per  se,  as  there  is  no   ^ 
standard  against  which  to  measure.   The  students  are  ON  THEIR 
HONOR  to  do  the  test  by  themselves  without  reference  material 
or  other  help.   The  test  must  be  turned  in  by  the  end  of  the 
working  day. 

(d)  Should  some  choose  to  take  the  test  then 

ajid  there,  position  yourself  in  the  room  so  that  you  are  avail- 
able to  answer  and  record  questions  the  participants  ask  of 
you.  Do  NOT  walk  up  and  down  the  aisles.   This  is  in  an 
effort  to  relaa  the  tensions  many  people  have  when  they  sit 
down  to  take  any  kind  of  a  test. 

(e)  Remember,  there  is  no  time  limit,  so 
allow  yourself  adequate  time  for  proctoring  the  test.   For 
planning  purposes,  the  test  is  designed  so  that  even  with  no 
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previous  background  in  human  factors,  the  test  should  be 
easily  completed  within  ^5  minutes.   Students,  however,  are 
not  required  to  complete  the  test  within  this  time  period, 

2,   Command  Course  Supervisor  Log:   The  CCS  Log  consists  of 
the  following  I 

SIFs  for  all  participants 

Student  Progress  Records 

Student/CCS  Questions  and  Comments  Record 

Unanswered  Student  Questionnaires 

CCS  Questionnaire 

Evaluation  Completion  Checklist 
FOR  FURTHER  CLARIFICATION t 

a.  SIFi   Keep  these  in  the  appropraiate  section  of  your 
log.  Dividers  have  been  provided.   See  also  the  "SIF  Instruc- 
tion" sheet, 

b.  Student  Progress  Records i   Record  the  completion  date 
fori 

(1)  SIFs 

(2)  Diagnostic  (D-Test)  Test  #1 

(3)  Each  lesson  book 

(4)  Diagnostic  Test  #2 

(5)  Student  Questionnaire 

(6)  CCS  Questionnaire 

Be  sure  to  keep  track  of  the  proper  SIF  number  for  each  student, 

c .  Student  Questions  and  Comments  Record »   Each  time 
either  you  or  a  student  have  a  question  or  comment,  record 
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the  following  I 

(1)  date. 

(2)  SIF  number. 

(3)  the  question  or  comment. 
(^)  the  answer  provided. 

d.   Student  Questionnaires!   Blank  questionnaires  have 
been  provided  so  that  they  are  available  as  each  student 
completes  the  second  diagnostic  test.   These  may  be  taken 
away  for  completion. 
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EVALUATION  OUTLINE 

I.   Purpose  of  the  Evaluation!   The  purpose  of  this  evaluation 
is  to  assess  the  operational  effectiveness  of  the  Human  Factors 
Engineering  (HFE)  Self- Instruction  Course  and  its  potential 
value  for  use  in  the  Fleet  test  and  evaluation  community. 

II.   Course  Descriptiont   This  course  is  divided  into  three 
major  sections,  Section  I  (lessons  1-5)  desils  with  the  human's 
capabilities  and  limitations.   In  this  section,  the  history 
and  continued  need  for  humain  factors  engineering  will  be  in- 
vestigated.  Lessons  6-10  are  concerned  with  the  physiological 
capabilities  and  limitations  of  hiiman  beings.   Lessons  11-13 
discuss  how  the  proper  design  of  controls  and  displays  makes 
use  of  the  information  learned  in  the  previous  lessons.   Lessons 
1^19  take  a  look  at  the  human  being  as  he/she  interacts  with 
hisAer  environment,  followed  by  a  review  of  the  first  section 
in  lesson  20. 

The  primary  focus  of  Section  II  is  upon  the  human  as  he/she 
fits  into  the  system.   Lessons  21-27 »  investigate  the  role  of 
the  human  engineer  in  various  aspects  of  analysis,  such  as 
systems  auialysis,  cost  analysis,  task  analysis,  etc.   Lessons  28-29 
are  concerned  with  the  selection  and  training  of  personnel.   In 
lessons  30-3^»  the  student  will  receive  a  short  course  on 
experimental  techniques  and  statistical  concepts.   Finally, 
lesson  35 »  will  offer  a  review  of  Section  II. 

Section  III  is  entitled  "Human  Factors  in  the  Military".   In 
this  section,  lessons  36  and  37  focus  on  human  factors  organ- 
izations, documentation,  and   future  application. 
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In  order  to  give  the  students  a  practical  application  of 
what  has  been  presented  in  the  course,  lessons  38  and  39  will 
ask  him/her  to  work  on  a  'real  world'  problem.   Finally,  in 
lesson  ^0,  a  more  typical  review  of  the  entire  course  is  pre- 
sented. 

III.   £urpose  of  the  HFE  Course i   The  HFE  Self- Instruction  Course 
was  developed  to  satisfy  a  need  for  increased  awareness  and  more 
indepth  imderstanding  of  Human  Factors. 

The  Comptroller  General,  in  his  report  to  the  Congress  of 
the  U.  S.,  dated  January  29,  1981,  (PSAD-81-17) ,  entitled 
Effectivenss  of  U.  S.  Forces  Can  Be  Increased  Through  Improved 
Weapon  System  Design,  highlights  this  need  and  identifies  a 
deficiency  in  the  performance  of  various  weapons  systems  "be- 
cause the  Department  of  Defense  does  not  pay  enough  attention 
to  logistic  support,  human  factors  and  quality  assurance  during 
the  design  phase  of  the  acquisition  process.   These  problems 
deter  the  system's  effectivenss  to  defend  our  country  in  case 
of  war. 

GAG  therefore  makes  recommendations  to  improve  the  manage- 
ment and  planning  of  ownership  considerations  that  have  an 
impact  on  the  effectiveness  of  a  weapon  system." 

The  HFS  Self-Instruction  course  was  developed  in  order  to 
provide  a  basic  knowledge  of  human  factors  and  how  to  test  for 
its  effectiveness.   The  course  has  not  been  evaluated  or  used 
on  a  trial  basis  in  any  portion  of  the  Navy  prior  to  this  time. 
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IV.   Summary  of  2valuation  Steps:   It  is  anticipated  that  each 
of  the  40  lessons  will  require  on  hour  completion  time.   While 
the  course  developers  recommends  40  working  days  to  complete  the 
entire  course,  time  constraints  levied  necessitate  completion 
prior  to  3I  October  1981. 

(1)  Introduction  by  Command  Course  Supervisor  (CCS) 

(2)  Receive  Privacy  Act  Statement 

(3)  Fill  out  Student  Information  Form  (SIF) 

(4)  Receive  instructions  for  and  take  Diagnostic  Test  ^^1 

a.  student  supplement 

b.  lesson  booklet  1-5 

c.  applicable  references 

(6)  Return  each  lesson  booklet  as  it  is  completed  and  pick 
up  the  next  one  in  the  series,  until  all  40  lessons  are  completed 

(7)  Receive  and  take  diagnostic  test  number  2 
a.  return  test  in  sealed  envelope 

(8)  Fill  out  student  questionnaire 

a.  Indicate  whether  or  not  test  results  are  desired 

b.  Be  sure  SIF  number  is  visible  on  envelope 

c.  Return  questionnaire  to  CCS  in  sealed  envelope 
within  two  working  days  or  not  later  than  3I  October  1981,  which 
is  soonest. 

Should  you  experience  any  repeated  difficulty  or  delays  in 
locating  your  CCS,  feel  free  to  contact  me.   I  do  not  have  my  own 
phone,  however,  a  secretary  will  take  the  message  and  I  will 
get  back  to  you.   Please  remember  that  there  is  a  three  hour 
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time  difference  between  the  East  and  West  Coasts.   Your  I345  is 
our  1645  and  the  secretaries  leave  at  I63O. 


Lt.  Martha  M.  Fleming 
Naval  Postgraduate  School 
SMC  #1340 
Monterey,  California   93940 


Autovoni   878-2536 
Commercial:   408-646-2536 
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DIAGNOSTIC  TESTS  INSTRUCTIONS 

Purposes   The  purpose  of  the  first  diagnostic  test  is  simply  to 
see  how  much  you  know  about  human  factors  before  you  begin  the 
Human  Factors  Engineering  (HFE)  Self- Instruction  Course.   This 
is  why  we  call  it  a  diagnostic  test.  After  you  have  completed 
the  HFE  course,  a  second  diagnostic  test  will  be  given.   The 
purpose  of  this  test  is  to  see  how  successfully  the  HFE  course 
has  increased  your  awareness  of  the  various  facets  of  human 
factors. 

Guidelines »   In  order  to  perform  an  accurate  comparison  ainalysis  of 
the  two  tests,  there  are  several  things  which  we  must  ask  of  you 
while  taking  these  diagnostic  tests < 

(1)  Answer  the  questions  to  the  best  of  your  knowledge, 
do  not  get  amy one  to  help  you  or  explain  the  question  to  you.  On 
the  first  diagnostic  test,  you  are  not  expected  to  know  the 
answers.   We  hope  that  the  HFE  course  will  help  you  answer  the 
questions  on  the  second  test. 

(2)  Do  not  guess  !   Please,  if  you  understand  the  question, 
but  do  not  know  the  answer,  select  the  option  which  indicates 

that  you  don't  know.   We  realize  that  this  is  against  your  human 
nature  and  therefore  is  difficult  to  do,  but  we  ask  that  you  try. 

(3)  This  is  an  evaluation  of  the  HFE  Course,  not  of  the 
course  participants.   You  do  not  flunk  or  pass;  the  course  does. 
If  it  is  so  indicated  by  your  tests,  questions,  comments  and 
questionnaires,  the  course  will  return  to  Pacific  Missile  Test 
Center  for  rework.   Therefore  everything  you  have  to  say  about 
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the  course  will  help  us  make  sure  it  will  be  a  good  one  when  it 
"hits  the  streets". 

(4)  If  a  question  does  not  make  sense  and  needs  clarification, 
see  your  Command  Course  Supervisor  (CCS). 

(5)  Your  CCS  will  be  recording  your  questions  as  you 
ask  them.   If  you  have  a  question,  then  obviously  the  materials 
didn't  make  it  clear  enough  and   need  to  be  changed.   You  csin  help 
your  CCS  by  writing  your  question  down  —  don't  forget  to  put 
your  SIF  identification  number  on  it. 

(6)  Since  the  HFE  course  is  designed  to  fit  the  needs 
of  the  job  of  testing  and  evaluating  human  factors,  rate,  rank, 
grade  or  educational  level  should  not  make  a  difference.  Again, 
if  it  does,  the  course  materials  need  to  be  improved. 

(7)  Where  you  take  the  test  is  up  to  you.   Your  CCS  will 
provide  you  time  and  a  place  to  take  the  test,  giving  you  easy 
access  to  him/her  for  questions  and  a  place  where  you  will  not 
be  disturbed.   You  may  elect  to  take  the  test  back  to  your  desk 
or  office,  however,  remember  not  only  the  HONOR  SYST'E^l   of  para- 
graph number  one,  but  also  that  the  test  must  be  returned  to 
the  CCS  in  a  sealed  envelope  by  the  end  of  that  working  day. 

(8)  Should  circumstances  beyond  your  control  prevent  you 
from  finishing  the  course  materials,  you  still  musti 

a.  Notify  your  CCS 

b.  Take  diagnostic  test  #2 

c.  Fill  out  the  student  questionnaire 

Since  each  test  is  progressively  designed  you  will  still  provide 
valuable  data  by  doing  (b)  and  (c)  above. 
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(9)  Do  you  wsint  to  know  your  test  results?  You  may 
make  that  choice  while  answering  the  student  questionnaire.   Your 
CCS  will  not  have  the  results  as  the  tests  are  analyzed  by  the 
Evaluation  Director  at  the  Naval  Postgraduate  School.   They 
will  be  forwarded  to  you  at  the  end  of  the  evaluation  upon 
receipt  of  your  request. 

(10)  Do  you  have  any  questions?  If  so  address  them  to 
your  CCS  now. 

In  Summary: 

(1)  Do  the  best  you  can. 

(2)  Do  not  guess. 

(3)  The  course  is  evaluated,  not  the  participant. 

(4)  Questions  and  comments  will  be  recorded  in 
logs  maintained  by  your  command's  Course  Supervisor. 

(5)  Rate,  rank,  grade,  AFS,  educational  level  sind 
prior  experience  should  not  make  a  difference. 

(6)  Take  your  Diagnostic  Test  #1  where  you  want,  but 
turn  it  in  by  the  end  of  that  working  day. 

(7)  Once  you  start  the  course,  plan  on  taking  diagnostic 
test  #2  amd  filling  out  the  student  questionnaire.   Return  the 
test  by  the  end  of  that  working  day  and  the  questionnaire  within 
two  working  days. 

(8)  Want  your  test  results?  Tell  us  so  in  the  questionnaire. 

(9)  Questions?  Ask  the  CCS. 

♦♦THANK  YOU  for  participating  in  this  evaluation i  without  you 
it  wouldn't  be  happening. 
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COMiyiAND  COURSE  SUPERVISOR  (CCS) 

EVALUATION  COMPLETION 

CHECKLIST 

1.  Administrative  Wrap-Up 

a.  Be  sure  that  each  of  the  following  has  been  completed i 

(1)  SIP  identification  numbers  on  all  completed  items. 

(2)  SIP  for  each  participant. 

(3)  Student  Progress  Records. 

ik)   Student/CCS  Question  &  Comment  Record. 

(5)  Diagnostic  Tests  #1  and  #2  from  EACH  PARTICIPANT 
beginning  the  Human  Pactors  Engineering  Self "Instruction  Course. 

(6)  Student  Questionnaire  from  EACH  PARTICIPANT  begin- 
ning the  course. 

b.  Retain  some  way  to  identify  students  with  their  SIF 
identification  numbers.   This  will  be  necessary  should  any  partici- 
pamts  request  their  test  scores  or  evaluation  results.  We  do 

not  wish  to  know  them,  only  that  you  be  able  to  pass  on  the 
information  upon  its  receipt. 

2.  Return  the  follow»ing  together  in  one  package  by  AIR  MAIL  — 
PIRST  CLASS  —  NO  LATER  THAN  1  NOVEMBER  1981  : 

a.   CCS  Logs  auid  all  completed  copies  ofi 

(1)  Student  Information  Forms  (SIP) 

(2)  Diagnostic  Test  #1 

(3)  Student  Progress  Record 

(4)  Student/CCS  Question  and   Comment  Record 
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(5)  Diagnostic  Test  #2 

(6)  Student  Questionnaires 

(7)  CCS  Questionnaire 

3.   Return  the  following  together  in  a  separate  package  by 
PARCEL  POST— Boolcratei 

a.   All  course  materials  and  extra,  uncompleted  forms  and 
questionnaires i 


(1 
(2 

(3 
(^ 
(5 
(6 

(7 
(8 

(9 


Student  Supplement 

Boolclets  for  lessons  1-^0 

Course  references  (MIL-STDs  and  TECHMEMOs) 

SIPs 

Diagnostic  Tests  1  and  2 

Student  Progress  Records 

Student/CCS  Question  and  Comment  Records 

Student  Questionnaires 

CCS  Questionnaires 
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STUDENT  QUESTION  &  COMMENT  RECORD 


DATE 


SIF  # 


Student's  Question/Comment 


17 


ANNEX  G 


DIAGNOSTIC  PRETEST 
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DIAGNOSTIC  PRETEST  INSTRUCTIONS 

PURPOSE.   The  purpose  of  this  test  is  merely  to  identify  the  level  of 
your  familiarity  with  hioman  factors  engineering  BEFORE  you  take  the 
self-paced  course.   A  second  diagnostic  test  will  be  given  after  les- 
son 20.   If  the  course  fulfills  its  purpose,  the  level  of  your  human 
factors  engineering  familiarity  should  increase.   The  results  of  the 
second  test  should  support  this  theory. 

FORMAT.   This  prediagnostic  test  has  eight  pages  and  28  questions.   If 
the  test  is  designed  correctly,  it  should  require  less  than  one  hour  to 
complete. 

DESIGN.   There  are  several  different  types  of  questions: 

1.  Multiple  Choice  -  These  questions  are  identified  "by  such  terms 
as  "CHOOSE"  or  "SELECT".  One  or  more  answers  may  be  correct.  Identify 
your  choice  by  placing  an  "X"  in  the  space  provided. 

2.  Matching   The  word  "I-IATCH"  denotes  such  questions.   Only  one 
answer  should  be  chosen  for  each  term  on  the  left.   Some  of  the  phrases 

or  acronyms  on  the  right  may  be  left  unused;  don't  worry,  the  question  was 
designed  that  way. 

3.  Fill  in  the  Blank  -  Words  such  as  "NAME",  "IS  CALLED"  or  "LABEL" 
identify  these  types  of  questions.   Sentences  requiring  completion  are  also 
examples  of  "fill  ins".   You  should  not  need  to  use  more  than  5  words  to 
answer  those  questions. 

h.  Short  Answer  -  These  questions  are  identified  by  such  terms  as 
"EXPLAIN",  "DESCRIBE",  "DEFINE",  "WHY",  "WHAT,  and/or  ask  for  examples. 


177 


Be  brief  and  to  the  point,  but  remember,  that  someone  else  must  be  able  to 
understand  your  answer;  so  work  on  its  clarity. 

5.   Application  -  One  question  asks  you  to  redesign,  if  necessary, 
a  display  mechanism.   Should  you  decide  it  is  needed,  DRAW  the  redesigned 
mechanism  in  the  space  provided.   Remember,  however,  you  may  be  happy  with 
the  displays  as  they  are  presented.   Should  that  be  the  case,  do  nothing. 

TEST  VALIDITY.   It  is  essential  that  you  do  this  test  by  yourself,  without 
the  use  of  references  or  outside  resources.   To  do  so  will  affect  the 
validity  of  the  analysis  method  selected  for  this  evaluation. 

GUESSING.   If  you  want  to  guess,  go  ahead.   You  would  anyway,  even  if  I 
told  you  not  to  do  so. 

GRADING.   Keep  in  mind  that  the  test  is  being  graded,  not  you.   Should 
you  like  to  know  the  score  the  test  made,  you  will  have  the  opportunity 
to  let  us  know  at  the  end  of  the  evaluation  period. 
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DIAGNOSTIC  PRETEST 


MULTIPLE  CHOICE:   Choose  the  best  answer(s).   One  or  more  than  one  answer 

may  be  correct.   Place  an  'X*  by  your  choice(s). 

A.   The  overall  objective  in  taking  a  human  factors  engineering  course  is: 


1.  to  obtain  the  human  factors  engineering  background 
necessary  to  do  your  job  well. 


2.    to  understand  a  person's  specific  capabilities  and 
limitations. 


3.  to  understand  the  precise  military  procedures  involved 
in  a  human  factors  program. 

_4.  to  understand  the  importance  of  applying  the  concepts  and 
priorities  of  human  factors  engineering  in  your  job. 


B.    The  main  reason  for  poor  system  performance  is  human  error  which  is 
caused  by: 


1.  inadequate  consideration  of  human  performance  capabilities, 
skill  limitations  and  response  tendencies. 

2.  little  standardization  of  'controls',  (knobs,  levers,  etc.) 

_3.  the  fast  developing  pace  of  new  technologies  with  which 
the  human's  capabilities  cannot  keep  up. 

4.  people  with  too  little  education  and  too  few  skills  being 
brought  into  the  armed  services. 


5.  inadequate  human  factors  input  into  system  man-machine 
interface. 


C.   There  are  three  main  areas  of  the  human's  information  processing  capabilities, 
MATCH  the  terms  on  the  left  with  the  appropriate  example (s)  on  the  right. 


1.  selective  attention 

2.  physiological  processing 
limitations 


a.  hear  a  signal  above  20,000  CPS 

b.  interrupting  normal  procedures 
to  react  to  an  alarm  bell. 


3.  channel  capacities 


c.  listening  to  four  incoming 
ship  to  ship  and  two  secure 
voice  radio  messages  simultan- 
eously on  the  bridge  of  a  ship 

d.  breaking  out  in  a  cold  sweat 

when  required  to  perform  a 
specific  duty. 


D.  MATCH  the  sensory  modalities  with  the  related  stimulus  dimensions: 

1.  vision  1.  saltiness 

1.    audition  2.  hue 

3.  odor  3.  pure  tones 

4.  taste  4.  smell 


E.   EXPLAIN  WHY  each  of  the  following  areas  is  important  in  human  factors 
design  evaluation: 


1.  information  processing  capabilities: 


2.  sensory  modalities 


F.  The  science  dealing  with  measurement  of  the  physical  features  and  functions  of 
the  body  is  called . 

G.  The  term  which  refers  to  the  measurement  of  human  body  dimensions  in  a  fixed 
position  is  . 

H.   NAME  the  term  which  indicates  that  body  dimensions  are  determined  from 

body  positions  which  occur  with  movement.    


I.   EXPLAIN  \^Wi   body  dimensions  are  an  important  human  factors  consideration. 
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J.   LIST  TWO  examples  which  support  your  definition  on  the  importance  of 
body  dimensions   as   a  human  factors  consideration: 

1. 


2. 


K.   NAME  five  environmental  considerations  which  may  affect  a  human's  performance: 

1. 

2. 

3. 

4. 

5. 


L.   CHOOSE  which  phrase  deals  with  the  principle  of  anthropometric  design; 


1.  use  of  population  stereotype 

2.  sequence  of  use  analysis 

3.  5th  —  95th  percentile  humans 

4.  design  for  the  extreme  individual 


M.   CHOOSE  which  phrase  deals  with  the  principle  concerned  with  the  general 
design  layout. 

1.  sequence  of  use 

2.  length  of  use 

3.  vigilance  capabilities  of  the  operator 

4.  frequency  of  use 

_5.  function  of  a  component 


N.   NAME  FOUR  methods  of  gathering  human  factors  data  on  human  activities 

1. 

2. 

3. 

4. 

0.   What  is  the  purpose  of  link  analysis?   GIVE  AN  EXAMPLE. 


P.   Knobs  should  have  basic  standardized  functions.   MATCH  the  terms  on  the 
right  with  the  figures  on  the  left. 


Column 
(1)      (2) 


Column   (1) : 

a.  fractional  reaction 

b.  discrete  reaction 

c.  multiple  reaction 


Column   (2) 


d.  less  than  1  full  turn  and 
position  is  not  very  important 

e.  less  than  1  full  turn  where 
position  is  important 

f.  at  least  1  full  turn 


cv. 
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Q.   CHOOSE  which  of  the  combined  effects  of  temperature  and  hiimidity 
the  human  body  will  adapt  to  most  quickly: 


1.  extreme  heat  and  high  humidity 

2.  extreme  cold  and  high  humidity 

3.  extreme  heat  and  low  humidity 


4.  extreme  cold  and  low  hijmidity 


R.   LABEL  the  following  display  mechanisms  as: 

(1)  quantitative  OR  qualitative  AND 

(2)  static  OR  dynamic 


ttcATCR         i/eMr  /V(c 


rtcrr 


C.oi.b 


a. 

aire: 

raft 

'turn 

and 

b. 

automobile 

c.  s 

bank. 

indicator' 

teimerature 

g 

control 

(1) 

(1) 
(2) 

(1) 

(2) 

(2) 

c.  submarine  depth 
gauge 


S.   If  you  would  improve  the  designs  presented  in  the  figures  above,  SHOW  what 
the  recommended  design  change  would  look  like  in  the  space  provided  below: 
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DEFINE  the  term  noise: 


U.   There  are  eight  concepts  for  evaluating  control  design.   Control  coding 
is  one  of  these.   NAME  FOUR  of  the  six  factors  which  are  used  when 
evaluating  control  coding: 

1. 

2.  

3.  

4. 


V.   Another  of  the  eight  concepts  for  control  design  deals  with  direction  of 
control  movement.   Briefly  EXPLAIN  WHY  this  is  important: 


W.   When  do  the  greatest  performance  decrements  occur  during  a  watch-standing 
duty?   CHOOSE  the  best  answer(s): 

1.  first  30  -  60  minutes  of  the  watch 

2.  last  30  -  60  minutes  of  the  watch 

3.  last  30  -  60  minutes  before  lunch  break 

4.  workspace  temperature  is  between  75°F  -  82°F 

5.  several  short  rest  periods  are  given  between  watches 


X.   What  is  a  task  analysis?   EXPLAIN: 
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Y.   Why  is  a  Cask  analysis  so  very  important  in  the  field  of  human  factors 
design  and  evaluation? 


Z.   MATCH  the  acronyms  with  the  associated  terms: 

^1.  radiation  a.  PPM 

2.  frequency  b.  PSI 

3.  atmospheric  contamination  c.  RAD 

4.  pressure  d.  CPS/Hz 

^5.  intensity  e.  CLO 

6.  hearing  loss  f.  dB/PNDB 

7.  insulation  g.  PTS/TTS 


AA.  NAME  TWO  key  military  standards  (MIL  STDS) ,  handbooks  or  U.  S.  Army 

Human  Engineering  Laboratory  Technical  Memorandums  which  provide  guidelines 
and/or  criteria  for: 

1.   control  design  


2.  environmental  factors/ 

considerations 

3.  anthropometric  data 


BB.  NAME  ONE  key  military  standard  (MIL  STD) ,  handbook  or  U.  S.  Army  Human 

Engineering  Laboratory  Technical  Memorandum  which  provides  guidelines  and/ 
or  criteria  for: 

1.  display  design  

2.  aural  non-detectability  

3.  noise  limit  selection  

4.  control  color  coding  
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5.  vigilance 

6.  criticism  regarding 
stated  criteria  gov- 
erning human  exposure 
to  carbon  monoxide  (CO) 


You  have  now  completed  your  first  diagnostic  test.   Don't  feel  badly  if  you 
think  that  you  didn't  know  many  of  the  answers.   You  weren't  expected  to,  remem- 
ber? This  test  has  been  a  brief  introduction  as  to  what  you  can  expect  to 
learn  as  you  proceed  through  the  HUMAN  FACTORS  ENGINEERING  SELF- INSTRUCTION 
COURSE.   The  first  lesson  booklet  and  the  student  supplement  will  be  handed 
to  you  when  you  turn  this  test  in  to  your  COMMAND  COURSE  SUPERVISOR. 
GOOD  LUCK  !!!!!!    Hope  you  enjoy  the  course  and  THANK  YOU  for  your  partici- 
pation and  support. 
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DIAGNOSTIC  POST  TEST 
INSTRUCTIONS 

PURPOSE.   The  purpose  of  this  test  is  to  measure  your  familiarity 
with  human  factors.   Some  of  you  will  be  taking  this  AFTER  com- 
pleting the  Human  Factors  Engineering  Self-Paced  Course 
Others  of  you  in  the  "control  group"  will  be  taking  this  test 
without  taking  the  course.   Those  of  you  in  the  "control  group" 
will  be  providing  a  baseline  from  which  to  measure  the  "course 
group's"  test  results. 

FORMAT.   This  diagnostic  post  test  has  7  pages  and  29  questions. 
If  the  test  is  designed  correctly,  it  should  require  approximately 
one  hour  to  complete. 

DESIGN .   Eventually  this  course  will  prepare  you  to  develop  a  gen- 
eral human  factors  evaluation  outline.   This  requires  a  thorough 
familiarity  with  the  following  areas  of  human  factors  design  and 
evaluation: 

Test  objectives  Terms 

Criteria  Definitions 

Analytical  methods         Design  principles  and 
Resource  documents  concepts 

The  content  of  this  test  will  establish  your  readiness  for  course 
completion. 

There  are  several  different  types  of  questions: 

1.   Multiple  choice.   These  questions  are  identified  by  the 
term  "CHOOSE".   None,  one  or  more  answers  may  be  correct.   Identify 
your  choice  by  placing  an  'X'  in  the  space  provided. 
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2.  Fill-in- the-blank.   Words  such  as  'TiAME** ,  "LA3EL"  or  "LIST" 
identify  these  types  of  questions.   You  should  not  need  to  use 
more  than  five  words  to  ajiswer  these  questions. 

3.  Short  answer.   The  questions  are  identified  by  such  terms 
as  "EXPLAIN",  "DESCRIBE",  "DEFINE",  "'/VHY",  "WHAT"  and/or  ask  for 
examples.   Be  brief  and  to  the  point,  but  remember,  that  someone 
else  must  be  able  to  understand  your  answer;  so  work  on  its  clarity, 

k.      Application.   One  question  asks  you  to  redesign,  if  necess- 
ary, a  display  mechanism.   Should  you  decide  it  is  needed,  DRAW 
the  redesigned  mechanism  in  the  space  provided.   Remember,  how- 
ever, you  may  be  happy  with  the  display  as  it  is  presented. 
Should  that  be  the  case,  do  nothing. 

TEST  VALIDITY.   It  is  essential  that  you  do  this  test  by  yourself 
without  the  use  of  references  or  outside  resources.   To  do  other- 
wise will  destroy  the  validity  of  the  analysis  method  selected 
for  this  evaluation. 

GUESSING.   If  you  want  to  guess,  go  ahead.   You  would  anyway, 
even  if  I  told  you  not  to  do  so. 

GRADING.   Should  you  like  to  know  the  results  of  the  test,  you 
will  have  the  opportunity  to  let  me  know  at  the  end  of  the  evalu- 
ation period. 


I 
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DIAGNOSTIC  POST  TEST 

A.   The  first  item  of  interest  is  the  TASK  ANALYSIS 
1.   When  should  a  task  analysis  be  done? 


2.   3y  whom  should  the  task  analysis  be  doneV. 


3.   Describe  what  information  the  task  analysis  provides. 


I   3.   SELECTION  OF  PERSONNEL  for  participation  in  each  test  system 
evaluation  is  an  area  of  critical  importance. 
1.   Explain  why  you  agree  or  disagree  with  this  statement. 


2.  Name  two  resource  documents  that  will  aid  you  in  selection 
of  test  subjects:   a. b. 

3.  3riefly  describe  the  basic  characteristics  of  a  typical 
test  subject: 


C.   WORKSPACE  DESIGN  AND  ARRANGERIENT  is  the  third  area  of  concern, 
1.  Name  two  resource  documents  that  will  aid  you  in  evalu- 
ating the  physical  layout  of  a  workspace  (i.e.  cockpit  or 
bridge).   a. b. 
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2.  Name  the  analysis  technique  recommended  for  evaluating 
the  arrangement  of  components  within  a  workspace.  


3.  Name  the  aspect  of  evaluating  workspace  design  which  con- 
siders the  physical  characteristics  of  intended  human 
operators.  

4.  Name  the  two  human  body  dimensions  measured  when  evalu- 
ating the  workspace  design  and  arrangement. 

a.  

b.  

D.   Evaluating  CONTROL  PANEL  DESIGN  should  be  next  on  your  list. 
CONTROLS,  DISPLAYS,  and  AUDIO/VISUAL  ALARI/iS  are  items  which 
require  specific  attention. 

1.  Name  two  resource  documents  that  will  provide  valuable 
guidelines  for  evaluating  a  control  panel. 

a.  

b.  

2.  Name  the  analysis  method/technique  which  is  recommended 
for  determining  whether  the  controls  and  displays  are 
positioned  optimally:  

3.  List  the  guidelines  recommended  when  deciding  whether  to 
use  an  audio  alarm  or  a  visual  alarm.   (DO  NOT  name  re- 
source documents  —  list  the  actual  guidelines. ) 

a.  

b. 


4.  List  six  of  the  eight  CONCEPTS  for  evaluating  CONTROL  DSSIGi^ 

a.  ^ 

b.  

c.  

d.  

e.  

f.  

5.  Select  one  of  the  concepts  from  question  4  and  name  four 
factors  considered  when  evaluating  this  concept, 
goncept^   a. 

factors >      (1)  

(2)  

(3)  

(4)  


6.   Select  another  one  of  the  concepts  listed  in  question  ^ 

and  explain  how  this  concept  could  be  critical  in  preventing 
human  errors. 

concept:   a.  

explanation;  


7.   There  are  three  classes  of  Imobs,  (1)  discrete,  (2)  fraction- 
al, and  (3)  multiple  reaction.   Figure  1  shows  four  knobs. 
Select  one  knob  for  each  of  the  controls  displayed  on  the 
next  page  and  place  its  corresponding  number  in  the  blanks 
provided. 
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HEATER 

IGNITION   SWITCH 

RADIO  FRE^UEInCY 

OFF 
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orp 

•<  ON 

n 

1       1        1        1        1        r        1       1       1   — 
So    to    to     So     1e     IM    M*    iLO   !)• 

• 

a. 


b. 


c. 


Choice  of  knobs: 


8. 


(4) 


Figure  1. 


If  and  only  if  you  believe  it  is  necessary  to  do  so, 
redesign  the  dial  in  the  space  provided  following  recom- 
mended design  practices.   Put  in  all  major  and/or  numbered 
markers.   Minor  markers  need  be  illustrated  only  between 
the  first  two  major  or  numbered  markers.  (Figure  2) 


ORIGINAL  DESIGN  NEW  DESIGN 

Figure  2.   Tachometer  used  in  testing  rotary  equipment.   The 
scale  unit  is  -5  rpm.   The  operational  range  is  5'^  rpm. 
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E.  The  INFORMATION  PROCESSING  CAPABILITY  of  the  human  operator 

is  a  critical  aspect  of  design.   Even  though  all  the  controls, 

displays  and  alarms  meet  their  requirements,  the  operators 

may  still  experience  difficulty  in  performing  their  necessary 

tasks.   These  difficulties  may  be  caused  by  the  limitations 

on  human  information  processing  capabilities.   There  are 

three  main  areas  in  which  these  capabilities  may  be  grouped. 

One  of  these  is  PHYSIOLOGICAL  PROCESSING  LIMITATIONS  (PPL). 
Give  an  example  for  PPL  and  name  the  other  two  main  areas: 

1.  PPL  example:  

2.  second  area:  

3.  third  area:  

F.  Weapon  systems  function  in  real  world  environments,  not  in 
vacuums.   The  operator  may  therefore  encounter  a  variety  of 
ENVIRONMENTAL  FACTORS  which  may  effect  his/her  performance. 

1.  Name  one  resource  document  which  provides  guidelines  in 
the  area  of  environmental  factors  and  their  possible  effects 
on  human  performsince.   

2.  Name  five  environmental  factors  and  for  each  factor  ex- 
plain what  effect  it  could  have  on  human  performsLnce; 

factor  effect 

a.   

b.   

c.   

d.   

e.  
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G.   There  are  ENVIRONMENTAL  CONSIDERATIONS  which  need  to  be  eval- 
uated.  Three  of  these  are  (1)  ATMOSPHERIC  CONDITIOInS,  (2)  iN'OISE, 
and  (3)  WORK/REST  SCHEDULES. 

1.   Define  the  term  noise* 


2.   Give  an  exaimple  of  an  atmospheric  condition  which  is  a 
concern  to  and  can  be  affected  by  a  human  factors  evalu- 
ator:         


3.   When  do  the  greatest  performance  decrements  occur  during 
a  watch- standing  duty? 


a. 
b. 


k.      Listed  below  are  UNITS  OF  MEASURE.   In  the  space  provided 
name  what  each  of  these  measures,  i.e..  in. /length;  lb./ 
weight. 

a.  CPS/rfz  

b.  PSI  

c.  CLO  

d.  PTS/TTS  

e.  RAD  

f.  d3/PNDB  

g.  PPM  
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H.   Choose  the  best  answer(s).   Place  an  "X"  by  your  choicels). 
One  or  more  than  one  answer  may  be  correct. 

1.  The  overall  objectiveCs)  in  taking  a  human  factors  engineer- 
ing course  is(are): 

1.  to  obtain  the  humaji  factors  engineering  background 

to  do  your  job  well. 

2.  to  understand  a  specific  person's  capabilities  and 

limitations. 

3.  to  understand  the  precise  military  procedures  in- 
volved in  a  human  factors  program. 

k.    to  understand  the  importance  of  applying  the  concepts 

and  priorities  of  human  factors  engineering  in  your 
job. 

5«  to  be  able  to  answer  some  questions  on  this  #*I^7fJ  test. 

2.  The  main  reason  for  poor  system  performance  is  human  error. 
Which  (if  any)  of  the  following  are  contributing  factors? 

1.  inadequate  consideration  of  human  performance  capa- 
bilities, skill  limitations  and  response  tendencies. 

2.  little  standardization  of  'controls',  (knobs,  levers,  etc.) 

3'  "the  fast  developing  pace  of  new  technologies  with 

which  the  human's  capabilities  cannot  keep  up. 

4.  people  with  too  little  education  and  too  few  skills 

being  brought  into  the  armed  services. 

5*  inadequate  human  factors  input  into  system  man- 
machine  interface. 
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POSTSCRIPT 

This  evaluation  was  originally  designed  to  cover  the  entire 
course,  lessons  1-40  over  a  period  of  60  days.  Murphy's  Law 
was  rolling  full  steam,  however,  and  the  evaluation  had  to  be 
cut  to  30  days  and  only  20  lessons.   You  all  are  the  ones  who 
lose  out,  though,  because  the  last  20  lessons  deal  with  quantify- 
ing human  factors  evaluations  and  practicing  human  factors 
concepts. 

Recently,  the  Government  Accounting  Office,  the  Secretary  of  De- 
fense and  others  have  pointed  out  that  the  military  is  accepting 
weapons  systems  that  can't  fulfill  their  missions.   They  say 
that  as  a  result  we  are  not  ready  to  meet  our  military  obligations. 
Human  error  is  pointed  out  as  being  one  of  the  prime  reasons  for 
the  unacceptable  system  performance.   They  also  state  that  inad- 
equate attention  to  human  factors  design  during  test  and  evaluation 
has  set  up  the  operator  to  make  the  error. 

If  we,  the  operational  test  directors,  know  what  design  factors 
can  cause  the  human  to  make  errors,  we  can  help  prevent  them. 
Perhaps  I'm  a  bit  surrealistic,  but  I  always  thought  our  job  was 
to  deliver  good  and  safe  equipment  to  our  military  men  and  women 
and  in  doing  so  help  save  lives. 

This  human  factors  course  may  have  been  an  assignment  to  you  — 
maybe  even  one  you  didn't  like.   Whether  you  wanted  to  take  the 
course  or  not,  I  sincerely  hope  you  have  learned  something.   If 
you  have,  then  the  course  is  serving  its  purpose  and  I  ask  you  to 
please  continue  with  the  other  20  lessons.   Some  flyer  or  sailor 
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will  be  out  there  thanking  you  for  it.   Who  knows,  it  may  even  be 
you. 

I    The  course  may  not  be  perfect  and  it  is  for  that  reason  the  evalu- 
ation is  being  conducted.   The  questionnaires  you  will  be  given 
is  your  opportunity  to  tell  me  how  to  make  it  better.   I  ask  for 
your  continued  dedication  and  co-operation  while  filling  out  the 
questionnaires.   The  feedback  you  provide  will  be  invaluable. 

Once  again,  thank  you  for  your  help  and  participation. 
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COMMAND  COURSE  SUPERVISOR 
QUESTIONNAIRE  INSTRUCTIONS 

Directions t   Based  upon  your  observations  and  your  log 

entries. -during  the  last  four  weelcs»  complete  this  form. 

Upon  completion,  submit  to  the  Evaluation  Director  at  the 

Naval  Postgraduate  School  for  analysis.   This  Evaluation 

Director  may  ask  for  your  assistance  and  for  clarification 

during  the  Analysis  Phase  of  this  evaluation. 


NAME COMMAND. 

NUMBER  OP  STUDENTS DATE 


SAMPLE  QUESTION. 

(3)  How  frequently  did  students'  questions/comments 
indicate: 

0-^0^  kl-60%   61-100^ 

a.  they  did  not  like 

coffee.  

b.  dissatisfaction 

with  today's  weather.   


If  half  of  all  students'  questions/comments  relating  to  cof- 
fee indicated  they  didn't  like  it,  then  place  aji  'X'  in  the 
4i-6Q^  column,  as  shown  below. 

If  there  are  nine  students'  questions/comments  relating  to 
today's  weather,  and  three  are  negative,  then  place  an  'X' 
in  the  O-^C^  column  (3  4-  9  =  33.3^) »  as  shown  below. 

0-40^  kl'60%   61-100^ 
a.   they  did  not  like  W 


coffee. 

b,   dissatisfaction 

with  today's  weather. 


X 


♦♦There  are  4  pages  to  this  questionnaire,  be  sure  to  amswer  all. 
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-  COMMAND  COURSE  SUPERVISOR 

'  QUESTIONNAIRE 

1.   How  frequently  did  students*  questions/comments  indicate 
dissatisfaction  with  thei 

(a)   LESSON  BOOK  MATERIALS.      0-i^O^  ^l-60^   6l-10O?5 

(1)  information  frames     

( 2 )  wrong  answer  frames    

(3)  questions  at  end  of 

information  frames     


» 


(4)  answer  choices  at 
end  of   information 
frames 

(5)  correct  answers 

(6)  course  length 

(7)  format 

(8)  lesson  length 

(9)  educational  level 
of  course  content 

(10)  course  content 

(11)  usefulness  of  the 
course 

(12)  amount  of  time 
spent  on  course 

(13)  terminology 

(b)   STUDENT  SUPPLEMENT* 

(1)  supplement  in  general 

(2)  graphs 

(3)  charts 
(^)   syrabology 

(5)  legends 

(6)  lesson  outlines 
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How  frequently  did  students'  questions/comments  indicate 
dissatisfaction  with  thei 

(c)  RESOURCE  DOCUMENTS!         0-40?S  ^l-60^   61-100% 

(1)  documents  (in  general) 

(2)  1^720  

(3)  1^7^B  

(d)  PRETEST! 

(1)  pretest  (in  general)  

(e)  POST  TEST 

(1)  post  test  (in  general) 


3.   How  frequently  did  students*  questions/comments  indicate 
that  initial  attitudes  about  human  factors  were! 

(a)  positive 

(b)  negative  

( c )  neutral  


^,   How  frequently  did  students*  questions/comments  indicate 
that  final  attitudes  about  human  factors  were! 


(a) 

postive 

(b) 

negative 

(c) 

neutral 

5.   How  frequently  did  students'  questions/comments  indicate! 

(a)  an  irritation  with 

course  materials.        

(b)  that  the  course  provided 

nothing  new.  

(c)  that  human  factors  train- 

ing was  unnecessary.     

(d)  that  the  learning  mater- 

ials were  more  hinder- 
ing than  helpful.        

(e)  that  the  course  was  a 

a  waste  of  time.         
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STUDENT  INFORMTION 

6.  How  many  students  failed  to  complete  20  lessons? 

7.  How  many  students  continued  beyond  lesson  20? 

8.  How  many  students  completed  exactly  20  lessons? 

9.  How  mamy  students  requested  to  keep  the  Student 
Supplement? 

10,   How  many  students  requested  to  teep  the  entire 
course? 


ADDITIONAL  COMMENTS t 
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STUDENT 

jUESTIONNAIRS 

INSTRUCTIONS 

PURPOSE:  The  main  purpose  of  this  questionnaire  is  to  ob- 
tain information  regarding  the  objectives,  content,  design 
sind  presentation  of  the  Human  Factors  Engineering  (HFE) 
Self-Paced  Course.  Your  answers  will  help  to  deter- 
mine what  actions  must  be  taken  to  improve  the  course  and 
the  quality  of  humsin  factors  training  and  education.  Your 
honest  opinions  are,  therefore,  essential. 

INSTRUCTIONS;   I  have  no  need  to  know  who  you  are  personnally. 
No  effort  will  be  made  to  identify  you.   However,  for  anal- 
ysis purposes,  it  is  necessary  to  have  your  Student  Identi- 
fication number  (SIF).   No  one  besides  myself  will  see  these 
questionnaires  and  no  individual  information  will  be  related 
to  your  command. 

Before  you  begin,  your  Command  Course  Supervisor  (CCS)  will 
show  you  an  example  of  an  "information  frame**  and  a  "wrong 
answer  frame" .   This  will  provide  clarification  of  terminol- 
ogy used  within  this  questionnaire. 

Directions  for  answering  the  questions  are  provided  below: 

SAMPLE  QUESTION 

(3)  What  types  of  schools         Types  of 

Schools 
have  you  attended?  Circle        1.  pre-school 

2.  elementary 
your  answer(s).  3*  junior  high 

k.    senior  high 
12  3^5  5-  ^-yr  high 
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If  you  attended  elementary  school,  you  should  circle  the 
number  2.  as  has  been  done  below,  since  the  number  2 
corresponds  to  elementary  school.   If,  in  addition,  you  also 
attended  a  ^--year  high  school,  you  should  also  circle  the 
number  5.  as  it  corresponds  to  4-year  high  school. 


I  0  3     '^   (f) 


SMPLE  QUESTION. 

(4)   In  taking  college  courses,  the  college  preparatory 

courses  taken  in  your  high  school  were 

very  effective 

effective 

borderline 

ineffective 

very  ineffective 

If  you  felt  the  college  preparatory  courses  you  took  in 
high  school  did  absolutely  nothing  to  help  you  through 
college,  place  an  'X*  in  front  of  "very  ineffective",  as 
shown  below: 

very  effective 

effective 

borderline 

ineffective 


X   very  ineffective 


If  you  have  any  questions,  please  ask  your  CCS  for  assistance. 

The  questionnaire  must  be  returned  to  your  CCS  within  two 

working  days.   3e  sure  to  double  check  that  you  have  answered 

all  questions. 

207 


HUr^N  FACTORS  ENGINEERING  SELF-INSTRUCTION  COURSE  03JSCTI7ES 

Upon  completion  of  the  Human  Factors  Engineering  Course,  the 
student  will  demonstrate  the  following  capabilities  and  knowledge: 

1.  An  understanding  of  common  terms  used  in  human  factors  engineering. 

2.  A  familiarity  with  human  factors  references  and  an  ability 
to  use  them. 

3.  An  awareness  of  potential  sources  of  technical  information  on 
major  human  factors  topics. 

^.   An  understanding  of  the  goals  of  human  factors  engineering  in  a 
materiel  development  program. 

5.  An  ability  to  integrate  human  factors  principles  in  a  DOD 
sponsored  program. 

6.  An  ability  to  determine  human  performance  requirements  in  a 
systems  concept. 

7.  An  under stsinding  of  the  kinds  of  factors  and  forces  which  affect 
human  performance  and  am  ability  to  identify  and  measure  them. 

8.  An  awareness  of  the  differences  between  field  and  laboratory 
measurements. 

9.  An  awareness  of  what  "experimental  control"  measures  are  necessary 
for  any  test  involving  human  performance  and  the  effects  in 
their  absence. 

.0.   An  understanding  of  basic  statistical  techniques,  such  as 
analysis  of  variance. 

An  ability  to  calculate  human  performance  reliability. 

An  ability  to  formulate  performance  measures  for  the  dependent 
variables  of  time  and  error. 

An  ability  to  analyze  human  performance  data  within  the  context 
of  "system  effectiveness"  and  "system  reliability." 

An  understanding  of  the  major  techniques  used  by  human  factors 
specialists  during  system  synthesis,  design,  and  development, 

A  familiarity  with  task  analyses. 

An  awareness  of  the  relationship  between  human  factors  engineering 
and  the  engineering  specialists  of  reliability,  maintainability 
and  safety. 

An  ability  to  interpret  and  apply  the  standards  and  specifications 
of  the  human  factors  engineering  community. 
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STUDENT 
QUESTIONNAIRE 

A.  Course  Content  and  Ob.iectivea. 

Refer  to  the  "Course  Objectives"  sheet  on  the  next  page  and 
answer  the  following  by  circling  your  answer(s). 


I 


(1)  Which  of  the  17  objectives  listed  have  been  supported 
(taught)  by  the  lessons  within  the  first  half  of  the 
Human  Factors  Engineering  Self-Instruction  CourseV 


1   2   3   ^   5   6   7   8   9  10  11  12  13  1^  15  16  17 

(2)  Which  of  the  17  objectives  listed  describe  a  skill  neces- 
sary to  test  and  evaluate  a  system's  human  factors  aspects? 

1   2   3   ^   5   6   7   3  9  10  11  12  13  1^  15  16  17 

(3)  Which  of  the  17  objectives  listed  would  help  you  do  your  job? 
1   2   3   ^   5   6   7   8   9  10  11  12  13  14  15  16  17 

B.   Course  Design 

(1^  The  sequence  in  which  the  various  lesson  topics  were 
presented  was 

very  effective 

effective 

borderline 

ineffective 

very  ineffective 


(2)  The  way  in  which  each  concept  was  developed  was 

very  effective 

effective 

borderline 

ineffective 

very  ineffective 


(3)  Direction  were  given  to  read  the  information  presented 
in  each  frame,  answer  the  question  at  the  end  of  each  frame 
and  go  to  the  page  referenced  by  the  selected  answer.   If 
the  answer  was  incorrect,  the  student  was  referred  back  to 
the  previous  page  to  select  another  answer.   If  the  answer 
was  correct,  the  student  proceeded  with  the  next  "information 
frame" .   This  format  was 

very  effective 

effective 

borderline 

ineffective 

very  ineffecxive 


Comment: 
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(4)  How  frequently  was  the  re- enforcement  or  reteaching 
provided  in  the  "wrong  answer  frame"  sufficient  to  choose 
the  "correct"  answer? 

always  frequently  sometimes  seldom  never 


(5)  How  frequently  was  the  answer  to  the  question  at  the 
end  of  each  "information  frame"  suggested  or  provided  in 
preceding  course  material? 

always 

frequently 

sometimes 

seldom 

never 


(6)  How  frequently  was  the  answer  to  the  question  at  the 
end  of  each  "information  frame"  suggested  or  provided  in 
course  material  following  the  question? 

always 

frequently 

sometimes 

seldom 

never 
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(7)  The  quality  of  the  questions  at  the  end  of  each  "infor- 
mation frame"  was 

excellent 

good 

borderline 

bad 

very  bad 


(8)  The  quality  of  the  answer  choices  for  each  question  at 
the  end  of  each  "information  frame"  was 

excellent 

good 

borderline 

bad 

very  bad 


C.   Course  Presentation 

(9)  Participation  in  the  HFE  course  without  the  aid  of  an 
on-site  instructor  was 

very  effective 

effective 

borderline 

ineffective 

very  ineffective 
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(10)  How  frequently  was  assistance  required  of  the  Command 
Course  Supervisor? 

always  frequently  sometimes  seldom  never 


(11)  How  frequently  were  words  used  which  required  defining 
by  an  outside  source  (dictionary,  person,  etc.)? 

always  frequently  sometimes  seldom  ^never 


I 


D.   Student  Supplement 

(12)  How  effective/ineffective  were  each  of  the  elements 

listed  below  in  aiding  your  understanding  of  the  concepts 

presented  in  the  course  material? 

Very 
Very  In-     In- 

effec-  Effec-  border-  effec-   effec- 
tive   tive    line    tive    tive 

tables  

charts  

graphs  

illustrations  

symbology  

legends  

terminology  

lesson  outlines 


arrangement 
of  tables, 
graphs,  etc. 
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(13)  How  do  you  feel  about  the  student  supplement  as  a  ref- 
erence book  and  human  factors  guideline  during  test  and  eval- 
uation planning  stages? 

extremely  useful 

of  considerable  use 

of  use 

not  very  useful 

of  no  use 


{m-}     The  information  given  with  figures  and  tables  (charts 
and  graphs)  to  provide  clarity  and  meaning  to  each  was 

extremely  useful 

of  considerable  use 

of  use 

not  very  useful 

of  no  use 


(15)  The  implied  ideas  and  meanings  represented  by  symbols 
and  legends  in  the  student  supplement  were 

always  understood 

frequently  understood 

sometimes  understood 

seldom  understood 

never  understood 
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E.   Resource  Documents 

(16)  The  HFE  course  materials  made  use  of  the  resource 

documents  listed  below 

very 

very  border-  diffi-  diffi- 

easy    easy  line    cult    cult 

1472B     

lk7^B  

HDBK  759  


(17)   Use  of  the  resource  documents  listed  below  in  planning 

human  factors  tests  for  a  new  system  is 

very 
very  border-  diffi-  diffi- 

easy    easy    line    cult    cult 

1^723     

147^3     

HDBK  759  


F.   Human  Factors  Attitudes 

(18)  How  do  YOU  FEEL  about  including  a  human  factors  test 
in  the  evaluation  of  a  new  piece  of  equipment  or  system? 

very  important 

important 

borderline 

unimportant 

very  unimportant 
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G.   Miscellaneous 

(19)   The  adventures  of  Lt.  I.  W.  Eager  were  tracked  through- 
out the  HFE  course.   The  stories  about  Lt.  Eager  were 

always  effective 

frequently  effective 

sometimes  effective 

seldom  effective 

never  effective 


COMMENT: 


(20)  How  frequently  did  the  factors  listed  below  adversely 
affect  your  completion  of  the  lessons  in  the  first  half  of 
the  course? 


always  frequently  sometimes  seldom    never 


Experience 

Education 

Time 

Job  load 

Course  Length 

Format 

Terminology 

Design 

TAD  (temporary 
additional  duty). 


(**)  Please  identify,  in  the  space  below,  any  specific 
problems  you  had  with  the  followingi 


LESSONS  BOOKS: 


STUDENT  SUPPLEIv:ENTj 


RESOURCE  DOCUI^ENTS: 


PRETEST: 


217 


POST  T2ST: 


(**■*)  Would  you  like  to  keep  a  copy  of  the  KFE  course? 

yes    no 

(***)  Would  you  like  to  keep  a  copy  of  the  student  supplement? 

yes    no 

(***)  Would  you  like  to  know  your  test  scores? 

yes    no 

(**•*)      Would  you  like  to   see   the   final   report   on  the  HFE 
Self-Faced  Course? 

yes         ^no 


WHAT  IS  YOUR  Student  Identification  Number? 
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COURSE   C3J2CTIVES   QUESTIONNAIRE 


I 
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COURSE  OBJECTIVES  EVALUATION 


NAT^IE 


COMAND 


PCS IT I ON /TITLE 


RESPONSIBILITIES 


EXPERIENCE  IN  COURSE  EVALUATIONS  (yrs.)  (mos.  ) 

TIME  AT  THIS  COMMAND  (yrs.)  (mos.) 

The  six  characteristics  listed  on  page  one  have  been 
established  by  NAVSDTRA  106A  as  criteria  by  which  in- 
structional learning  objectives  are  to  be  judged.   The  17 
learning  objectives  listed  on  pages  two  and  four  will, 
therefore,  be  evaluated  according  to  how  completely  they 
meet  the  six  criteria  as  explained  on  the  next  two  pages. 
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Directions:  Assess  each  learning  objective  according  to 
each  of  the  six  criteria.   Place  criteria  numbers  (1»2,3,^,5» 
and  6)  in  the  spaces  to  the  right  of  each  learning  objective, 
according  to  how  completely  each  objective  satisfies  each 
of  the  criteria.   Each  criteria  should  be  rank  ordered  from 
"Not  at  All"  to  "Completely".  ys>     y^^   /  ^^ 

SAMPLE: 


A.  A  student  will  be  able  to 
correctly  solve  quadratic 
equations. 


CRITERIA  FOR  COURSE  OBJECTIVES 

1.  Objectives  must  be  a  statement  of  student  behavior 
(action),  such  as  the  creation  of  a  product  or  some 
other  overt  act,  which  can  be  accepted  as  evidence  that 
the  intended  outcome  has  occurred. 

2.  The  behavior  must  describe  specifically  all  outcomes 
that  will  demonstrate  that  learning  has  occurred. 

3.  The  student  behavior  called  for  must  be  capable  of 
observation  and  evaluation  within  the  learning  and 
testing  environments. 

4.  The  objective  must  be  stated  in  learner  rather  than 
"teacher"  terms,  i.e.,  actions  which  the  student  will 
perform  rather  than  what  the  instructional  materials 
will  "say  or  do". 

5.  There  must  be  a  standard  against  which  the  student 
behavior  will  be  measured.   It  must  be  fully  specified. 

6.  The  statement  of  the  conditions  under  which  the  student 
behavior  will  occur  must  be  fully  specified. 
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If  you  feel  that  objective  "A"  fulfills  criteria  1,2,3, 
and  k   completely,  then  place  the  numbers  1.2.3.  and  U-   in  the 
blank  under  "Completely"  as  shown  below.   If  you  feel  that 
objective  "A"  does  not  meet  criteria  5  at  all,  place  a  "5" 
under  "Not  at  All".   If  it  meets  criteria  6  to  some  extent, 
but  needs  quite  a  bit  of  attention,  then  place  a  "6"  under 
"To  Some  Degree",  as  show  below. 


A.  A  student  will  be  able  to 
correctly  solve  quadratic 
equations. 


If  you  have  any  questions  or  comments  regarding  this 
evaluation,  please  feel  free  to  contact  me  or  write  them 
in  the  space  provided  at  the  end  of  this  evaluation. 


Lt.  Martha  M.  Fleming 
Naval  Postgraduate  School 
SMC  ^   1340 
Monterey,  California   93940 

Autovon:   873-2536 
Commercial  J   408-646-2536 
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LEARNING  OBJECTIVES 


1.  An  understanding  of  cominon  terms 
used  in  human  factors  engineering. 

2.  A  familiarity  with  human  factors 
references  and  an  ability  to  use 
them. 

3.  An  awareness  of  potential  sources 
of  technical  information  on  human 
factors  topics. 

M-.      An  understanding  of  the  goals  of 
human  factors  engineering  in  a 
material  development  program. 

5.  An  ability  to  integrate  humsin 
factors  principles  in  a  Department 
of  Defense  sponsored  program. 

6.  An  ability  to  determine  human  per- 
formance requirements  in  a  systems 
concept. 

7.  An  understanding  of  the  kinds  of 
factors  and  forces  which  affect 
human  performance  and  an  ability 
to  identify  and  measure  them. 

8.  An  awareness  of  the  differences 
between  field  and  laboratory 
measurements. 

9.  An  awareness  of  what  "experimental 
control"  measures  are  necessary  for 
any  test  involving  human  perform- 
ance and  the  effects  in  their  ab- 
sence. 

.0.   An  understanding  of  basic  statis- 
tical techniques,  such  as  analysis 
of  variance. 

An  ability  to  calculate  human  per- 
formance reliability. 

An  ability  to  formulate  perform- 
ance measures  for  the  dependent 
variables  of  time  and  error. 
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CRITERIA  FOR  CQURS£  QBJECTIV5S 

1.  Objectives  must  be  a  statement  of  student  behavior 
(action),  such  as  the  creation  of  a  product  or  some 
other  overt  act,  which  can  be  accepted  as  evidence 
that  the  intended  outcome  has  occurred. 

2.  The  behavior  must  describe  specifically  all  outcomes 
that  will  demonstrate  that  learning  has  occurred. 

3.  The  student  behavior  called  for  must  be  capable  of 
observation  and  evaluation  within  the  learning  and 
testing  environments. 

^.   The  objective  must  be  stated  in  learner  rather  than 
"teacher"  terms,  i.e.,  actions  which  the  student  will 
perform  rather  than  what  the  instructional  materials 
will  "say  or  do". 

5.  There  must  be  a  standard  against  which  the  student 
behavior  will  be  measured.   It  must  be  fully  specified. 

6.  The  statement  of  the  conditions  under  which  the  student 
behavior  will  occur  must  be  fully  specified. 


NAVSDTRA  IO6A 
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LEARNING  OBJECTIVES 


13.  An  ability  to  analyze  human  per- 
formance data  within  the  context 
of  "system  effectiveness"  and 
"system  reliability". 

14.  An  understanding  of  the  major 
techniques  used  by  human  factors 
specialists  during  system  synthesis, 
design,  and  development. 

15.  A  familiarity  with  task  analyses. 

16.  An  awareness  of  the  relationship 
between  human  factors  engineering 
and  the  engineering  specialists  of 
reliability,  maintainability  and 
safety. 

17.  An  ability  to  interpret  sund  apply 
the  standards  and  specifications 
of  the  human  factors  engineering 
community. 


A.   Are  these  the  criteria  by  which  you  judge  learning  objectives? 
(Circle  one):  YES         NO 

If  the  above  answer  was  "NO",  please  provide  a  copy  of  the 
criteria  you  do  use. ) 

3.   Do  you  approve  of  these  6  characteristics  being  used  as 
criteria  for  judging  learning  objectives?   (Circle  one): 

YES         NO 

C .   Why? : 
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APPENDIX  F 
DEMOGRAPHIC  AND  PRE -POST  TEST  DATA 


\ 
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APPENDIX  H 

STUDEiNT  ATTITUDE  SURVEY 
(QUESTIONNAIRE) 
DATA 
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Table  28 
Record  of  Student  Attitude  (Questionnaire)  Responses 

A.   COURSE  CONTENT  AND  OBJECTIVES 

1.  Objectives  taught  in  the  first  20  lessons: 

Obj.  #:      1    2    3    4    5    6    7    8    9    10 
Responses:   20   19   11   16   10   11   14    7    7    2 

Obj.  *:      11   12   13   14   15   16   17 
Responses:    2    3    2   10   10    6    8 

2.  Objectives  necessary  to  do  HFE  Test  and  Evaluation: 

Obj.#:  1         2        3        4         5        6         7        8        910 

Responses:         887799      10         597 

Obj.  #:      11   12   13   14   15   16   17 
Responses:   11   10   12    5    9    8    6 

3.  Objectives  which  would  help  you  do  your  job: 

Obj.#:       1    2    3    4    5    6    7    8    910 
Responses:   10   12   11    9   10    9   12    6    9    8 

Obj.^:      11   12   13   14   15   16   17 
Responses:    8    7   13    9    9   13    6 
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Table  28  -  Continued 

B.   COURSE  DESIGN 

Student  Responses 

Most  Most 

Positive  Negative 

5     4      3      2      1 
I  I  I  I /    Total 

1.  Sequence  of  les- 
son topics  2     14     4     1     3      24 

2.  Concept  develop- 


ment 

1 

13 

5 

2 

3 

24 

3. 

Format 

1 

11 

2 

5 

5 

24 

4. 

Remediation 

1 

6 

9 

3 

4 

23 

5. 

Info  provided 
before  questions 

1 

11 

9 

2 

1 

24 

6. 

Info  provided 
after  questions 

2 

5 

14 

1 

2 

24 

7. 

Question  quality- 

1 

11 

10 

1 

1 

24 

8. 

Answer  choice 
quality 

1 

10 

10 

1 

2 

24 

C.       COURSE    PRESENTATION 


9.    Course   w/o   on-site 

instructor  18  744  24 


10.  Freq.    use   of   CCS        15  6  3  -  -  24 

11.  Terminology--def s . 

provided  6    9      7     2     -      24 
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Table  28  -  Continued 
D.   STUDENT  SUPPLEMENT 


Student  Responses 

Mo  s  t  Mo  s  t 

Positive  Negative 

5     4      3     2      1 

I  I  I  I  I  Total 


12.  Effectiveness  of  course  elements 


Tables 

2 

15 

6 

- 

- 

23 

Charts 

2 

14 

7 

- 

- 

23 

Graphs 

2 

13 

8 

- 

- 

23 

Illustrations 

2 

IS 

6 

- 

- 

23 

Symbo  logy- 

- 

12 

8 

2 

1 

23 

Legends 

- 

14 

7 

1 

- 

22 

Terminology 

- 

15 

6 

- 

2 

23 

Lesson  outl 

ines 

3 

10 

8 

1 

1 

23 

Arrangement 
tables  5 

of 
graphs 

_ 

12 

7 

3 

1 

23 

13.  Usefulness  of  Stu- 
dent Supplement  as 
a  reference         -     6    10     7     1      24 


14.  Student  Supplement 
(usefulness  %   self- 
explanatory)  --fig- 
ures %    tables       -      8     13      3     -       24 


15.  Clarity  of  symbols 

and  legends         4     11      7     1      1      24 
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Table  28  -  Continued 
E.   RESOURCE  DOCUMENTS 


Student  Responses 

Most  Most 

Positive  Negative 

5     4     3     2     1 

I  f  I  I  I  Total 


16.  Course  made  reference  documents  easier/harder  to  use 

1472C  2  8      7     2      -      19 

1474B  2  7      5     2      -      16 

HDBK  759  2  11      5     2      -       20 


17.  Use  of  ref  documents  in  planning  HFE  T  5  E  made  easier/ 
harder: 

1472C  -  10     6      3     -      19 

1474B  -  8      6     3     -      17 

HDBK  759  -  11      5     3     -       19 


F.  HUMAN  FACTORS  ATTITUDES 

18.  Is  HFE  T  5  E  necessary: 

12    10     1     1     -      24 

G.  MISCELLANEOUS 

19.  Story  Line  Effectiveness: 

1      3     10     4     6      24 
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Table  28  -  Continued 

G.   MISCELLANEOUS  (cont'd) 

Student  Responses 
(Never)  (Always) 

Mo  s  t  Mo  s  t 

Postive  Negative 

5      4      3      2      1 
I  f  I  I  I  Total 

20.   Factors  Adversely  affecting  completion  of  lessons: 


Experience 

10 

5 

5 

3 

- 

23 

Education 

10 

4 

5 

3 

1 

23 

Time  Available 

1 

1 

9 

9 

3 

23 

Job  Load 

1 

-- 

7 

11 

3 

22 

Course  Length 

6 

6 

5 

4 

2 

23 

Format 

3 

6 

5 

1 

9 

24 

Terminology 

8 

2 

9 

5 

- 

24 

Design 

5 

4 

3 

4 

5 

22 

Temporary  Addi- 
tional Dury 

6 

3 

8 

5 

2 

24 

H.   COURSE  DESIRABILITY 

Yes  No  Total 

Students  want  to  keep  course  materials.   10  14    24 

Students  want  to  keep  Supplement  only.    13  11     24 
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APPENDIX  I 

QUESTIONNAIRE  DATA 
FROM  CURRICULUM  DEVELOPMENT  EXPERTS 
ON  TERMINAL  LEARNING  OBJECTIVES 
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Figure  1 
Module  Cover  Page  Format 


MODULE  1.0 

HUMAN  CAPABILITIES  AND  LIMITATIONS 

FOR 
HUMAN  FACTORS  ENGINEERING 


Course  Identification  Number 


Prepared  by- 
Pacific  Missile  Test  Center,  Pt.  Mugu,  California 

25  December  1983 
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Figure  2 
Module  Overview  Format 


MODULE  OVERVIEW 
MODULE  1.0 

Human  Factors  Engineering 
HUMAN  CAPABILITIES  AND  LIMITATIONS 


In  this  module 


List  of  Lesson  Topics  Included  in  Module: 

Lesson  Topic  1  -  Welcome  to  Human  Factors  Engineering 
Average  Time :   

Lesson  Topic  2  -  Why  Human  Factors? 
Average  Time:   

Lesson  Topic  3  -  Tragic  Mistakes ...  and  Positive 

Consequences . 
Average  Time:   


Lesson  Topic  20  -  Review 
Average  Time:   


Total  Module  Average  Time 


(Insert  any  general  instructions  to  students  concerning  the 
use  of  this  Module  Booklet  and  any  of  its  supporting  mater- 
ials . ) 


244 


I 

I 


« 


Figure  3 
Lesson  Topic  Cover  Page  Format 


HUMAN  FACTORS  ENGINEERING 

MODULE 

NUMBER 

1.0 

LESSON 

TOPIC 

1.1 

WELCOME 

TO  HUMAN 

FACTORS 

ENGINEERING 

25  December  1983 
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Figure  4 
Lesson  Topic  Overview  Format 


MODULE  1.0 
LESSON  TOPIC  1.1 


LESSON  TOPIC  OVERVIEW 
LESSON  TOPIC  1.1 
WELCOME  TO  HUMAN  FACTORS  ENGINEERING 


In  this  Lesson  Topic  . . . 


The  Learning  OBJECTIVES  of  this  Lesson  Topic  are  as 
follows : 

1. 

2. 


4. 


(Statement  to  the  student  that  he  or  she  should  review  the 
"LIST  OF  STUDY  RESOURCES"  and  read  the  Lesson  Topic  LEARN- 
ING OBJECTIVES  before  beginning  the  Lesson  Topic.) 
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Figure  5 
List  of  Study  Resources  Format 


MODULE  1.0 
LESSON  TOPIC  1.1 


LIST  OF  STUDY  RESOURCES 
WELCOME  TO  HUMAN  FACTORS  ENGINEERING 


To  learn  the  material  in  this  LESSON  TOPIC,  you  have  the 
option  of  choosing,  according  to  your  experience  and 
preferences,  any  or  all  of  the  following  study  resources. 

Written  Lesson  Topic  presentations  in  the  Module  Booklet: 

1.  Lesson  Topic  Summary 

2.  Programmed  Instruction  Form  of  Lesson  Topic 

3.  Student  Supplement  containing  supporting 
information,  charts,  graphs,  etc. 

4.  Lesson  Topic  Progress  Check 

5.  Narrative  Form  of  Lesson  Topic 

Additional  Materials 

1.   Student  Response  Sheets 

a.  Diagrams,  charts,  graphs 

b.  Programmed  Instruction  Response  Sheets 

c.  Answer  Sheet  for  use  with  all  tests 

d.  Notetaking  Sheets 

Enrichment  Materials 

1.  Additonal  Resource  Documents  from  the  Student 
Packet 

2.  Additonal  References  from  the  Naval  Supply 
System 

You  may  use  any  or  all  resources  listed  above,  including 
the  Learning  Course  Sponsor,  but  all  materials  listed  are 
not  necessarily  required  to  achieve  Lesson  Topic  Objectives 
The  Progress  Check  may  be  taken  at  any  time. 
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Figure  6 
Lesson  Topic  Summary  Format 


MODULE  1.0 
LESSON  TOPIC  1.0 


LESSON  TOPIC  SUMMARY 


Welcome  to  Human  Factors  Engineering 


(A  condensation  of  the  narrative  form  of  the  lesson  topic 
is  placed  here) . 


Statement  to  the  student 


"At  this  point,  y 
Check.   If  you  answer 
ceed  to  the  next  Less 
only  a  few  of  the  Pro 
Answer  page  will  refe 
graphs,  or  frames  so 
Lesson  Topic  you  are 
that  you  have  failed 
Lesson  Topic,  select 
Narrative  or  consulta 
until  you  can  answer 
Check  correctly." 


ou  may  take  the  Lesson  Topic  Progress 

all  self-test  items  correctly,  pro- 
on  Topic.   If  you  incorrectly  answer 
gress  Check  Questions,  the  Correct 
r  you  to  the  appropriate  pages,  para- 
that  you  can  restudy  the  parts  of  this 
having  difficulty  with.   If  you  feel 
to  understand  all,  or  most,  of  the 
and  use  another  medium  of  instruction: 
tion  with  the  Learning  Course  Sponsor, 
all  self-test  items  on  the  Progress 
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